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Introduction 


Exactly twenty years ago, in July 1999, Nature Neuroscience, then a recently 
launched scientific journal, published a manuscript entitled “Real-time control of a 
robot arm using simultaneously recorded neurons in the motor cortex.” The result of 
one more collaboration between my laboratory and John Chapin’s, this paper launched 
the field of brain-machine interfaces (BMIs) in earnest and caused a major sensation in 
the neuroscience community. Curiously, the name brain-machine interface would only 
appear a year later (Nicolelis, 2001), coined in a review paper entitled “Actions from 
thoughts” that I wrote for Nature, following a request by one of its editors, Charles 
Jennings, who happened to be the first editor-in-chief of Nature Neuroscience. 

To celebrate the twenty-year anniversary of that landmark paper, I have decided 
to collate all BMI-related articles published by my lab at the Duke University Center for 
Neuroengineering, as well as the Edmond and Lily Safra International Institute of 
Neuroscience and the neurorehabilitation laboratory that I founded in Brazil during the 
past two decades. The aim of this initiative is to place in two volumes, the first of a 
series entitled the Nicolelis Lab Series, the entire library of manuscripts that my students 
and I have produced as a result of our research in the BMI field. To some degree, these 
volumes are also historic documents that show the evolution of our ideas in the area as 
well as the many “spin offs” that emerged from our tinkering with the original — and 
now classic — concept of BMIs. That included our transition from using this approach 
primarily as a basic science tool to probe the brain, in search of the physiological 
principles governing the behavior of neuronal circuits — or neural ensembles, as I like 
to call them — to our current major drive to develop assistive and even therapeutic 
strategies for neurological and psychiatric disorders that incorporate BMIs, either alone 
or in conjunction with other technologies (e.g. virtual reality and robotics) and clinical 
approaches. 

To help seasoned practitioners and newcomers alike, I have divided the 
manuscripts that form these two volumes in multiple categories, which loosely cover 
some of the history of the BMI field, at least the way I see it, while offering a reasonable 
way to reconstruct how the rational and experimental work performed in my labs 
evolved until it reached its current focus on developing multiple BMI-based therapies. 

Following this basic structure, Volume 1 begins with a collection of manuscripts 
that I deem the “Foundation” papers. This section contains some of the key studies 
executed, initially when I was a postdoc in John Chapin’s lab, and then by my own lab 
at Duke University that led to the maturation and optimization of the fundamental 
experimental methods employed in the invention and dissemination of BMI during its 
first fifteen years (Nicolelis et al., 1995). I am referring to the neurophysiological 
approach known as chronic, multi-site, multi-electrode recordings in behaving animals, 
which John Chapin and I originally developed and implemented in rats, between 1989 


and 1993, and which I lately adapted and scaled up with the goal of carrying out studies 
with New and Old World non-human primates. This shift from rodents to primates 
was pivotal for our lab’s ability to follow up the original Nature Neuroscience study 
with the first demonstration of a BMI in primates, published a year later, in Nature 
(Wessberg et al., 2000). 

The next section in Volume 1, Essential Methods, contains a large list of 
manuscripts that describe in detail all the key neurophysiological, biomedical 
engineering, computational, robotics, and behavioral approaches that played a central 
role in the development of all BMIs implemented in my labs. This section is followed 
by clusters of papers that summarize our experimental studies in rodents and primates. 
Volume 2 begins with the collection of all our clinical BMI studies to date. The next 
two sections of Volume 2 are dedicated to three special BMI spinoffs that were 
originally proposed and developed at great length by our lab at Duke University. I am 
referring to the creation of sensory neuroprosthesis in rodents (our “infrared rat 
project”), and two paradigms at the edge of the field: brain-to-brain interfaces and 
brainets. In the latter section, I include a recent manuscript that reports on the first 
demonstration of simultaneous, multi-brain, multi-channel recordings based on wireless 
technology (Tseng et al., 2018). This is particularly relevant because, in the future, this 
approach will allow neurophysiologists to perform animal social studies while 
concomitantly recording the brain activity from large numbers of untethered subjects 
interacting among themselves. By the same token, this approach will allow the concept 
of “shared BMIs,” 1.e. a BMI operated by the simultaneous contributions of multiple 
individual brains engaged in the collective performance of a behavioral task. 

The final two sections of Volume 2 contain a series of opinion and review papers, 
including a very comprehensive and recent review of the entire field of BMI, including 
its historic origins, and potential future clinical applications (Lebedev and Nicolelis, 
2017). 

Revising all this material brought me a lot of great memories and rekindled some 
of the tremendous “academic battles” John Chapin and I had to fight in the beginning 
to convince some of our more recalcitrant and conservative colleagues in systems 
neuroscience that BMI was not only a legitimate experimental tool to probe the brain, 
but that, in due time, it could also become a major approach to assist and treat patients 
suffering from a variety of brain disorders that still today challenge clinicians, given the 
lack of suitable therapies to ameliorate them. Fortunately, the vast majority of these 
colleagues, given the evident success of the field and its clear potential benefits, have 
graciously changed their minds to become enthusiastic supporters of this quickly 
expanding field of inquiry. A few, sadly, still resist, calling BMI only an “applied 
science,” as if this is an offense to those of us who took the field, in twenty years, from 
its basic science origins, all the way to clinical studies that promise to unleash the full 
potential of this paradigm to millions of patients. 


Since our public demonstration of the potential impact of BMIs during the 
opening ceremony of the 2014 Soccer World Cup in Brazil, when a paraplegic patient 
was able to use a non-invasive BMI to deliver the opening kickoff of that event for a 
worldwide TV audience estimated at 1.2 billion people, the public fascination with 
BMIs, and its potential impact for the improvement of patients suffering from brain 
disorders, has skyrocketed all over the world. 

Having witnessed, many times over, the enthusiastic reaction of scientific and lay 
audiences alike all over the planet, when I show them that those humble rat studies, 
carried out in John Chapin’s Philadelphia lab in the early 1990s, allowed us, merely 
twenty years later, to offer a paraplegic young man, paralyzed from the mid-chest down, 
a way to walk again only by his own thinking, I can only say that it was all worthwhile. 

By a long shot. 


1995 


1998 


1999 


2000 


2003 


2004 


2007 


2009 


A014 


2013 


2014 


2015 


Time Table of Key Discoveries 


First demonstration of chronic, multi-site, multi-electrode recordings in 
freely behaving rodents (Nicolelis et al., Science 1995). 


First chronic, multi-site, multi-electrode recordings in non-human primates 
(Nicolelis et al., Nature Neuroscience, 1998). 


First BMI study using rodents (Chapin et al. Nature Neuroscience 1999). 


First BMI for non-human primates (Wessberg et al., Nature 2000). 
Introduction of the name “brain-machine interface” (Nicolelis Nature, 2000). 


First BMI for reaching and grasping in Rhesus Monkeys (Carmena et al. 
PLOS Biology 2003). 


First invasive multi-channel BMI in humans (Patil et al., J. Neurosurgery 
2004). 


First BMI for bipedal walking using monkey brain activity to control a 
humanoid robot (Cheng et al., Soc. Neurosci. Abstract 2007). 


A new neuroprosthesis for Parkinson’s disease based on spinal cord electrical 
stimulation (Fuentes et al., Science 2009). 


First brain-machine-brain interface for active touch exploration using a 
multichannel cortical micro-stimulation (O’Doherty et al., Nature 2009). 


First BMI for bimanual control (Ifft et al, Science Translation Medicine, 2013) 
First brain-to-brain interface (Pais-Vieira et al., Scientific Reports 2013). 

First sensory neuroprosthesis for infrared sensing (Thomson et al., Nature 
Communications 2013). 


First wireless multi-channel BMI in non-human primates (Schwarz et al., 


Nature Methods 2014). 


First brainet for non-human primates (Ramakrishnan et al., Scientific Reports 


2015). 


2016 First demonstration that chronic BMI use leads to neurological recovery in 
spinal cord injury patients (Donati et al., Scientific Reports, 2016). 


2018 First simultaneous, wireless, multi-subject brain recordings in monkeys during 
performance of a social motor task (Tseng et al., 2018). 


Clinical Brain-Machine Interface Studies in 
Parkinsonian and Spinal Cord Injury Patients 
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NEUROSURGERY 


| CLINICAL STUDIES 


ENSEMBLE RECORDINGS OF HUMAN SUBCORTICAL 
NEURONS AS A SOURCE OF MOTOR CONTROL SIGNALS 
FOR A BRAIN-MACHINE INTERFACE 


OBJECTIVE: Patients with severe neurological injury, such as quadriplegics, might 
benefit greatly from a brain-machine interface that uses neuronal activity from motor 
centers to control a neuroprosthetic device. Here, we report an implementation of this 
strategy in the human intraoperative setting to assess the feasibility of using neurons in 
subcortical motor areas to drive a human brain-machine interface. 

METHODS: Acute ensemble recordings from subthalamic nucleus and thalamic motor 
areas (ventralis oralis posterior [VOP]/ventralis intermediate nucleus [VIM]) were 
obtained in 11 awake patients during deep brain stimulator surgery by use of a 
32-microwire array. During extracellular neuronal recordings, patients simultaneously 
performed a visual feedback hand-gripping force task. Offline analysis was then used 
to explore the relationship between neuronal modulation and gripping force. 
RESULTS: Individual neurons (n = 28 VOP/VIM, n = 119 subthalamic nucleus) 
demonstrated a variety of modulation responses both before and after onset of changes 
in gripping force of the contralateral hand. Overall, 61% of subthalamic nucleus 
neurons and 81% of VOP/VIM neurons modulated with gripping force. Remarkably, 
ensembles of 3 to 55 simultaneously recorded neurons were sufficiently information- 
rich to predict gripping force during 30-second test periods with considerable accu- 
racy (up to R = 0.82, R? = 0.68) after short training periods. Longer training periods 
and larger neuronal ensembles were associated with improved predictive accuracy. 
CONCLUSION: This initial feasibility study bridges the gap between the nonhuman 
primate laboratory and the human intraoperative setting to suggest that neuronal 
ensembles from human subcortical motor regions may be able to provide informative 
control signals to a future brain-machine interface. 


KEY WORDS: Brain-machine interface, Neuroprosthesis, Single-unit recording, Subthalamic nucleus, 
Thalamus 


Neurosurgery 55:27-38, 2004 DOI: 10.1227/01.NEU.0000126872.23715.E5 www.neurosurgery-online.com 


H uman neuroprosthetic devices cur- 
rently depend on control signals from 
residual nerve or muscle activity to 
restore motor functions lost because of disease 
or trauma. It has been proposed that these 
devices could be significantly improved by 
directly harnessing brain activity from central 
motor-related regions to drive artificial actua- 
tors (6, 14, 17, 21, 22, 31). Recently, laboratory 
studies involving nonhuman primates have 
made considerable advances toward the de- 
velopment of such devices. For example, neu- 
ronal ensemble recordings from motor areas 
of cerebral cortex in nonhuman primates have 


been demonstrated to accurately predict 
three-dimensional arm movements (26, 27, 29) 
and to successfully control a robotic arm neu- 
roprosthetic device (5, 29). Despite these inter- 
esting advances, primate studies have yet to 
address the fundamental question of whether 
current brain-machine interface (BMI) tech- 
nology and approaches may be successfully 
applied to human patients (9, 21, 22). Single- 
unit neuronal recordings from human cerebral 
cortex have been used to drive cursor move- 
ment in a simple neuroprosthetic application, 
establishing feasibility for this approach (14, 
15). However, nonhuman primate BMI studies 
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suggest that multineuronal recordings are critical for neuro- 
prosthetic applications and may require a minimum of 50 to 
100 recorded neurons to drive a real-time neuroprosthesis (5, 
22). 

In addition to cortical motor regions, subcortical regions, 
such as the motor thalamus and subthalamic nucleus (STN), 
are also involved in motor planning and execution. Neuronal 
activity associated with upper-extremity movements has been 
observed in both the STN (1, 25) and motor thalamus (1, 11, 18, 
25). Both the motor thalamus and STN are also targets of deep 
brain stimulation for the treatment of Parkinson’s disease and 
tremor disorders (2), thereby providing a valuable opportu- 
nity to test this hypothesis directly in humans. 

Here, we hypothesized that ensemble neuronal activity in 
human subcortical regions may provide effective motor con- 
trol signals for task prediction and ultimately for neuropros- 
thetic control. To test this hypothesis, we designed a novel 
intraoperative visual feedback-associated hand motor task us- 
ing gripping force to drive cursor movement to a target. In 
addition, we designed and used a unique 32-microelectrode 
array for acute multineuronal ensemble recordings, allowing 
definition of a quantitative relationship between neuronal ac- 
tivity and applied hand force. Our method used linear and 
nonlinear decoding algorithms to test whether sufficient mo- 
tor control signals could be derived from these subcortical 
regions to predict task activity. Multiple individual neurons 
modulated firing with task performance, and ensembles of 
neurons provided excellent predictions of patient gripping 
force, suggesting that subcortical motor targets may provide 
robust control signals for BMI neuroprosthetic applications. 


PATIENTS AND METHODS 


Patient Selection and Localization of Subcortical 
Motor Centers 


Patients with either tremor (essential tremor or Parkinson’s 
disease with tremor dominance) or severe, refractory Parkin- 
son’s disease were selected for either thalamic (ventralis inter- 
mediate nucleus [VIM]) or STN deep brain stimulator (DBS) 
(Medtronics, Minneapolis, MN) placement. The study was 
approved and monitored by the Duke University Institutional 
Review Board. After providing informed consent, all patients 
were given the option to withdraw from the study at any time. 
There were no intraoperative complications during the study. 
All patients underwent postoperative computed tomographic 
scanning of the head. We observed no evidence of intraoper- 
ative hemorrhage or intracranial injury caused by either the 
recording sessions or DBS placement in any of the 
participants. 

For patients who consented to participate, up to 10 minutes 
of intraoperative recording time was used for these studies. 
All surgical procedures were performed through frontal burr 
holes using the Leksell stereotactic frame while patients were 
awake, after a localization magnetic resonance imaging scan 
was performed. Standard coordinates were used as the initial 
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targets for VIM (4-6 mm anterior to the posterior commissure, 
13-15 mm lateral from the midline, and at the anterior com- 
missure—posterior commissure line) and STN (3 mm behind 
the AC-PC midpoint, 12 mm lateral to the midline, and 4mm 
below the anterior commissure—posterior commissure line). 
The STN was identified physiologically by the cell density, the 
cell firing frequency, the characteristic length (5-6 mm) of 
neurons along the recording track, and the presence of kines- 
thetic cells (25). The VIM was identified by the presence of 
tremor cells and a location just anterior to the sensory nucleus 
of the thalamus. For the STN, the multichannel electrode was 
placed into the middle of the nucleus after the single-channel 
electrode was used to determine a track length of at least 5.5 
mm within the STN. For the motor thalamus, the first track 
was aimed posteriorly into the sensory nucleus, and the upper 
aspect of this track (8-10 mm from the sensory nucleus) was 
used for multichannel recording. This location used for mul- 
tichannel recording was either directly in or in close proximity 
to the ventralis oralis posterior (VOP). This location is there- 
fore identified as VOP/VIM. 


Motor Task Performance 


Patients were placed in a supine, semisitting position in 
front of a video screen monitor. A squeeze ball (Laboratory for 
Human and Machine Haptics, Massachusetts Institute of 
Technology, Cambridge, MA) in the hand contralateral to the 
recording electrode array measured gripping force. Once po- 
sitioned, the patient was trained to perform the gripping force 
task as described below, in Results. The patient was instructed 
to adjust hand-gripping force to reach a target level as rapidly 
as possible once the target was presented. Once at a target 
level, the patient was required to hold that level of force for a 
short period (0.5-1.0 s) before the target level would change. 
There was no alerting for the next target force level. The 
patient’s shoulder and arm were allowed to rest on an arm- 
board, so that the only detectable motion used for the task was 
gripping force. For many of the Parkinson’s disease patients 
(who were in the “off” state, not receiving medication), there 
was a considerable delay in reaction time before the force 
change to move to the new target location was initiated. 
Typically, patients could perform the task for 5 minutes con- 
tinuously before tiring, allowing up to 50 task repetitions for a 
single session while neural ensemble recordings were ob- 
tained. Presentation of video images, generation of the motor 
task, and recording of patient responses was performed by use 
of custom software written in C** by the first author (PGP). 


Electrode Arrays and Electrophysiological Recording 


Neuronal activity was simultaneously recorded during mo- 
tor task performance with either a standard single-channel 
microelectrode (5 wm, tungsten, 0.5 MO; FHC, Inc., Bowdoin- 
ham, ME) or a custom 32-channel platinum-iridium microelec- 
trode array (Fig. 1). The multichannel array was fashioned 
after that used for chronic single-unit recordings in hippocam- 
pal and mesial temporal lobe structures (10, 16) and was 
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FIGURE 1. Multichannel, multineuronal recordings in human subcortical 
nuclei. A, preoperative and intraoperative appearance of the multichannel 
array. Left, the array consists of a bundle of thirty-two 40-wm platinum- 
iridium microwires, individually insulated but within an outer sheath (bar 
= 1 cm). The array can span up to 4 to 5 mm when inserted to a depth of 
10 mm below its brain-protective 1.0-mm stainless steel guide-tube can- 
nula. Right, intraoperative lateral cranial fluoroscopy of the array in situ. 
The array is seen as diverging microwires within the concentric circular 
targets of the Leksell stereotactic frame, which is affixed to the patient's 
cranium. In the foreground, a contralateral deep brain stimulator (DBS) 
electrode (Medtronics, Minneapolis, MN) is observed. Each of the four 
DBS contact electrodes is 1.5 mm in length, for comparison. The 4-mm 
divergence of the microwires in the STN target region allows the microw- 
ires to sample the majority of the STN. B, simultaneously recorded neuro- 
nal activity from VOP/VIM with spike detection and sorting. Left, elec- 
trophysiological records from four individual microwires. Right, individual 
waveforms obtained after spike detection and sorting, illustrated on an 
expanded time base. Waveforms are color-coded to indicate the source 
channel and normalized by amplitude. Note that channel 12 records two 
well-differentiated neurons, whereas channel 19 demonstrates a single 
dominant signal with increased baseline noise, most likely because of 
increased neuronal density in the neighborhood of this electrode. 


organized as a coaxial but diverging bundle to avoid tissue 
injury (0.6-mm outer diameter). Intraoperative fluoroscopy 
(Fig. 1A) suggests that during implantation, the microwires 
diverged to approximately 4 mm, sufficient to cover most of 
the STN. This microwire array builds on the arrays used for 
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chronic recordings in nonhuman primates (23), but it is com- 
mercially available and approved by the United States Food 
and Drug Administration (FDA) for short-term recordings. 

The single-channel electrode was used to determine the 
optimal recording depth within a region containing a high 
density of neurons. The multichannel electrode array was then 
passed immediately afterward to the same depth (within 1 
mm). Although correlations between single-channel electrode 
recordings and patient movement were sought as part of the 
standard surgical procedure, correlations between movement 
and multichannel recordings with the array were not, because 
of time constraints. Hence, there was no measurement of a 
direct or specific relationship between active or passive hand 
movement and the multichannel recordings at the time of 
electrode placement. Given this functionally random anatomic 
localization of the multichannel array, we expected the array 
to record a mixture of responses to hand movement, with no 
response to movement in some neurons. Neuronal signals 
were recorded and stored at 40 kHz with a 32-channel per- 
sonal computer-based acquisition system (Plexon, Inc., Dallas, 
TX). Spike detection and spike sorting were performed as 
described previously (23), using a threshold-based criterion 
for spike detection and principal component analysis for spike 
sorting. 


Detection of Responsive Neurons 


Patient responses were sorted according to the change in 
target force as either increased force or decreased force. Peri- 
event histograms of neuronal activity were constructed sepa- 
rately for positive and negative force transitions by use of 
100-ms bins 0.5 to 1.0 second before and 2 to 3 seconds after 
the time of target force transition. Throughout our analysis, 
the reference (zero) time for each transition was aligned to the 
time at which the new target location was presented to the 
patient, which was defined more precisely in time than the 
initiation of patient movement. For each patient, the response 
time between change in target and change in applied force 
was similar across trials during the recording session and was 
therefore typically a simple offset from the reference zero 
time. Neuronal responsiveness to changes in gripping force 
was defined as changes in neuronal firing rate to beyond the 
95% confidence interval for baseline activity. To qualify as a 
responsive neuron, firing rates were required to modulate 
beyond the 95% confidence interval in both positive and neg- 
ative directions with opposite changes in force. 


Prediction of Motor Activity from Neuronal Activity 


Prediction of motor activity from neuronal recordings with 
a Wiener filter (4) requires approximately 10 minutes of con- 
tinuous data to determine the parameters of the model (5, 29). 
However, the conditions in which the current human record- 
ings were performed do not allow continuous recordings for 
more than 5 minutes. Hence, other linear and nonlinear meth- 
ods that require fewer data and less training time to generalize 
than the Wiener filter, such as a Kalman filter (12), normalized 
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least mean squares (LMS) (12), and feed-forward artificial 
neural network (ANN) (3) algorithms, were tested. Each 
model was typically trained using 3 minutes of data, with 
predictions made for subsequent 30-second epochs. Neuronal 
data were binned at 100 milliseconds. The Kalman filter was 
given an initial condition equal to the real initial condition and 
an a priori estimate of the error covariance set to 0.1; LMS was 
trained with 10 lags and a learning rate of 0.2; the ANN had 2 
layers (10 units hidden-layer) with tangent sigmoidal and 
linear transfer functions, respectively, and was trained using 
conjugate gradient descent with Powell-Beale restarts. In cases 
in which the output of the models contained high-frequency 
ripple, the signal was low-pass filtered with a fifth-order 
Butterworth filter (cutoff frequency, 2 Hz) and multiplied by a 
constant gain factor of 1.2 (all parameters were found exper- 
imentally). Analyses were performed in Matlab (The Math- 
Works, Natick, MA) using the Neural Networks toolbox (8). 
To predict motor activity at a given moment in time, only 
neuronal data from up to 1 second before that moment were 
used in the calculation. Neuronal firing after the moment of 
interest was excluded from analysis. This prevented the inad- 
vertent use of neuronal activity generated in response to the 
movement and limited calculations to neuronal activity asso- 
ciated with motor planning. 


RESULTS 


Ensemble Neuronal Recordings from Human 
Subcortical Motor Centers 


We recorded the electrophysiological activity of neuronal 
ensembles from the STN and thalamic motor regions (VOP/ 
VIM) in patients with Parkinson’s disease and essential 
tremor, respectively, during surgical placement of DBS de- 
vices (Fig. 1A). All patient selection and intraoperative studies 
were completed in accordance with the approval, policies, and 
stipulations of the Duke University Institutional Review 
Board. Microelectrode recording was performed with either a 
standard single-channel tungsten microelectrode or a 32- 
channel platinum-iridium microelectrode array. Figure 1A il- 
lustrates the 32-channel array, shown photographically before 
implantation and fluoroscopically within the STN during a 
recording session, opposite a contralateral DBS electrode that 
had just been placed. Note that the small microwires are 
barely visible but clearly show divergence and are close to the 
target area. 

The results presented here were obtained from 13 STN 
recording sessions in 8 patients (n = 119 neurons) and 8 motor 
thalamic (VOP/VIM) recording sessions in 3 patients (n = 26 
neurons). Between 3 and 55 neurons were recorded simulta- 
neously, depending on the composition of the signals mea- 
sured from each of the microelectrodes in the array. As illus- 
trated in simultaneously recorded traces in Figure 1B (from 
VOP/VIM), spike discharges from individual units varied in 
both morphology and frequency. Details of the sorted action 
potentials from single neurons obtained in simultaneously 
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recorded channels are shown to the right of the traces. Base- 
line recordings reflected both the observed noise level and the 
fact that high-quality single neurons could be acutely isolated. 
Neuronal activity was resolved by use of a threshold criterion 
for spike detection, whereas individual units were differenti- 
ated in a single microelectrode signal by principal-component 
analysis (5, 23). 

By use of these methods, high-quality neuronal units could 
be sensitively and specifically detected and differentiated (Fig. 
1B, right). In rare channels, neuronal discharges could not be 
reliably differentiated and were therefore analyzed as a single 
composite unit. These results confirm that a simple microwire 
recording array functions effectively to capture multiple, si- 
multaneous channels of neuronal data in vivo in human sub- 
cortical targets. The channels were independent, with varying 
spike frequency and spike morphology, because of the sepa- 
ration of the microwires, as confirmed by the intraoperative 
fluoroscopy (Fig. 1A). 


Ensemble Neuronal Activity during Motor 
Task Performance 


During each operative recording session, patients per- 
formed a novel hand-gripping task synchronously with phys- 
iological recording (Fig. 2). A visual representation of gripping 
force was provided to the patient as the location of a black 
vertical bar on a video monitor (Fig. 2A, inset). The target area 
into which the patient had to move the black bar was repre- 
sented by a green rectangle on the screen, which varied ran- 
domly in a time-dependent manner. The motor task required 
the patient to match the black bar to the green target by 
adjusting the degree of gripping force exerted on a hand-held 
pressure bulb. The response of the vertical bar movement to 
gripping force was scaled so that patients could reliably reach 
positions over the entire target range. The absolute amount of 
force required for full-range movement therefore varied 
among patients but was held constant during each recording 
session. Release of the pressure bulb caused the vertical bar to 
move off scale to the left. This constant maintenance of grip- 
ping pressure required the patient to remain attentive to the 
task. Our patients were able to perform the hand motor task 
for 3 to 5 minutes before tiring. Kinematic records of task 
performance reflect the delayed reaction time and baseline 
tremor of patients with Parkinson’s disease (black trace, Fig. 
2A). 

Figure 2B shows a 24-channel STN ensemble of neuronal 
activity represented by a raster plot, which was recorded 
simultaneously with the epoch of the behavioral task shown in 
Figure 2A. Consistent with previous studies (1, 11, 18, 25, 28), 
some neuronal units in both STN and motor thalamus seemed 
to respond to upper-extremity movement. By simultaneously 
recording both neuronal activity and force generation, corre- 
lation between neuronal activity and motor task performance 
could be observed and defined quantitatively. For the record- 
ing session shown in Figure 2, note that many of the neurons 
modulate with patient force production (Fig. 2B). 
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FIGURE 2. Simultaneous measurement of kinematic parameters and neuronal activity during perfor- 
mance of a motor task. A, patients tracked a randomly time-varying target on a video screen (inset) 
by applying variable gripping force to a pressure measurement device in the contralateral hand. The 
kinematic record illustrates target force (green) and patient gripping force (black) over time. Com- 
plete release of the device results in movement of the black cursor off-screen to the left. Hence, the 
patient was required to maintain attentiveness and positive gripping force throughout the session. 
The sensitivity of the pressure ball was adjusted to allow comfortable movement throughout the target 
range before each experimental session, and although linear, it was not recalibrated. Positive deflec- 
tion indicates increased gripping force. B, simultaneously recorded STN neuronal activity, in raster 
format, measured with a 32-channel multielectrode array. Note the correspondence of neuronal modu- 
lation to force production in this 24-unit recording. Time scale is the same as in A. 


(0.5-1 s) at each target force level, the 
average force records may not have 
sharply defined baseline levels and tran- 
sition times. However, the polarity of 
correlation and the phasic relationship 
between firing frequency and force are 
quite apparent. 

Two exemplar VOP/VIM neurons 
(Fig. 3A, left and middle) illustrate a neg- 
ative correlation of firing frequency and 
force in which changes in neuronal firing 
precede changes in force generation. By 
comparison, the modulation of the re- 
maining VOP/VIM exemplar neuron 
(Fig. 3A, right) also precedes changes in 
force generation but is positively corre- 
lated. By contrast, changes in STN firing 
also demonstrated positive (Fig. 3B, left 
and right) and negative (Fig. 3B, middle) 
correlation; however, the time course of 
STN modulation was slower, making de- 
termination of the temporal relationship 
between neuronal firing and force gener- 
ation more difficult to discern. 


Ensemble Neuronal Analysis and 
Task Prediction 


Studies in nonhuman primates dem- 
onstrate that neuronal ensemble record- 
ings from motor areas of cerebral cortex 
accurately predict three-dimensional arm 
movements (26, 27, 29) and are able to 


Quantitative Correlation of Neuronal Activity to Motor 
Task Performance 


Modulation of neuronal firing with gripping force was ex- 
amined in detail by constructing peri-event histograms of 
neuronal activity aligned to the time of target movement (Fig. 
3). Overall, 7 (27%) of 26 VOP/VIM neurons showed in- 
creased firing rates with increased force (positive correlation), 
whereas 14 (54%) of 26 VOP/VIM neurons showed decreased 
firing rates with increased force (negative correlation), as 
tested with 95% confidence limits around the baseline firing 
frequency. By comparison, 46 (39%) of 117 STN neurons 
showed positive correlation between firing frequency and 
force, and 24 (21%) of 117 STN neurons demonstrated nega- 
tive correlation. Hence, 81% of VIM/VOP neurons and 61% of 
STN neurons showed statistically significant modulation with 
hand-gripping force (19). 

Temporal relationships between neuronal modulation and 
changes in force are apparent through comparison between 
the timing of changes in neuronal firing, as represented by the 
histograms at the bottom of each panel in Figure 3, and the 
timing of changes in force, as represented by curves at the top 
of each panel. Note that because of relatively short epochs 
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successfully control a robotic arm neuro- 
prosthetic device (5, 29). To assess the suitability of ensembles 
of subcortical neurons as a source of motor control signals for 
a human BMI neuroprosthesis, we examined the ability of 
subcortical ensemble activity to predict patient gripping force. 
We estimated predictive ability by training linear (Wiener 
filter, Kalman filter, normalized LMS [12]) and nonlinear 
(feed-forward ANN [3]) algorithms with force and neuronal 
data during “training” periods of up to 300 seconds. These 
trained models and the neuronal activity during a subsequent, 
nonoverlapping “test” period were then used to predict grip- 
ping force during the test period. 

Despite severe constraints because of limited time, small 
neuronal ensembles, and operative and patient considerations, 
the recorded neuronal signals were sufficiently information- 
rich that the trained models were able to predict hand- 
gripping force during the test period quite accurately, as 
shown for four sessions (Fig. 4, A-D, R* = 0.38-0.68). Overall 
prediction performance across 20 recording sessions was sig- 
nificant, although modest (R? = 0.26 + 0.04; mean + standard 
error of the mean). In general, the ANN, LMS, or Kalman filter 
decoding algorithms outperformed the simpler Wiener filter 
algorithm. However, no single decoding algorithm demon- 
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FIGURE 3. Modulation of individual neurons in VOP/VIM and STN 
during task performance in six patients. A, VOP/VIM neurons exhibiting 
modulation with task performance. Each panel illustrates average patient 
gripping force production (top), neuronal firing in raster format (mid- 
dle), and a peristimulus time histogram relationship (bottom). The histo- 
gram is aligned so that target transitions to a new force level occur at 
time zero. Panels for increasing target force (above) and panels for 
decreasing target force (below) are displayed for each modulated neuron. 
Note that the VOP/VIM neuron represented in the left panel demon- 
strates an inverse correlation of firing with patient force production. The 
right neuron demonstrates the opposite. B, STN neurons exhibiting modu- 
lation with task performance; same format as in A. 


strated superiority over the others (data not shown). Further- 
more, similar to findings in nonhuman primate studies (29), 
prediction accuracy increased with both training period dura- 
tion (Fig. 4E) and the number of neurons within the ensemble 
(Fig. 4F), suggesting that the longer training intervals and the 
larger ensemble sizes possible with chronic implantation of a 
human BMI would probably result in improved BMI 
performance. 

These findings demonstrate feasibility for the hypothesis 
that chronically recorded neuronal units in subcortical motor 
centers may provide an effective motor control signal for task 
prediction. In nonhuman primate studies, model training pe- 
riods of 10 minutes are common, and then animal training 
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FIGURE 4. Prediction of gripping force from recorded neuronal activity 
by linear and nonlinear decoding algorithms. A—D, four examples of grip- 
ping force prediction from VOP/VIM and STN neuronal activity during 
task performance. A, prediction using ANN (4 VOP/VIM neurons, R = 
0.65 for ANN, R = 0.14 for Wiener). B, prediction using ANN (8 VOP/ 
VIM neurons, R = 0.62 for ANN, R = 0.48 for Wiener). C, prediction 
using LMS (26 STN neurons, R = 0.82 for LMS, R = 0.14 for Wiener). 
D, prediction using Kalman filter (4 STN neurons, R = 0.67 for Kalman, 
R = 0.35 for Wiener). E, increasing predictive quality with longer train- 
ing time (3 VOP/VIM neurons, Wiener method). Data are illustrated as 
mean correlation coefficient + standard error of the mean for 20 trials 
with randomly selected, nonoverlapping training and testing epochs. F, 
increasing accuracy of gripping-force prediction with larger neuronal 
ensembles. Explanatory power (R*) increased with the number of neurons 
when linear or nonlinear methods were used (blue). One data point is 
shown for each of the 20 recording sessions. In each case, the best predic- 
tion for the session is plotted. 


periods of several days may be needed for accurate task 
prediction. Our findings resemble the results of nonhuman 
primate studies (5, 29), because increasing correlations be- 
tween hand-gripping force and model predictions were ob- 
tained by adding neurons and lengthening the training period. 
Thus, the multichannel human data are highly promising for 
development of a chronic human multichannel BMI if the 
recordings can be extended to larger groups of neurons and 
the models remain useful for these chronic recordings. 


DISCUSSION 


Overall, our findings are novel in several important re- 
spects. This is the first demonstration of simultaneous record- 
ing of sizable numbers of human subcortical neurons, indicat- 
ing that microwire arrays (in some form) are likely to be 
feasible for use in a future human neuroprosthetic device (Fig. 
1). The individual neurons recorded and their response to a 
behavioral task seem to be similar to those observed by many 
groups, indicating that the multichannel electrodes are faith- 
fully recording appropriate signals from these subcortical mo- 
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tor regions (Figs. 2 and 3). The power of ensemble recordings 
over single-unit recordings is the ability to use the neurons 
together in a decoding algorithm to predict motor activity. In 
several sessions (Fig. 4), we found that models based on ex- 
tremely short training periods could accurately predict hand- 
gripping force. Such quantitative predictions are a far more 
stringent analysis than traditional single-unit correlations be- 
tween changes in neuronal firing characteristics and task per- 
formance. In addition, such predictions, calculated on neural 
and motor activity in a previous, nonoverlapping period, pro- 
vide the key to implement real-time brain-machine interfaces 
for neuroprosthetic and motor enhancement applications. Our 
study thereby provides translation of this conceptual under- 
standing from chronic, nonhuman primate recordings to the 
acute human intraoperative setting and suggests that ensem- 
bles of subcortical neurons may potentially serve as a source 
of motor control signals for a neuroprosthetic BMI in humans. 


Microwire Recording Electrodes 


Many different electrode designs have been suggested for 
the chronic ensemble recordings required of a neuroprosthetic 
BMI. These designs include silicon-based chip electrodes (20), 
the neurotrophic electrode (14, 15), and either fixed grid arrays 
of microwires (23) or loose microwires inserted as a bundle 
(10, 16). Among these, microwire arrays have been demon- 
strated to record chronically from multiple neurons for peri- 
ods up to 2 years (23). In addition, at the present time, none of 
the electrode arrays are FDA-approved or available, except for 
the neurotrophic electrode and a loose bundle of microwires. 
We therefore implemented a 32-channel version of these mi- 
crowire bundle electrodes. 

During recording sessions, multiple channels demonstrated 
excellent performance in either the STN or VOP/VIM (Fig. 1) 
on a short-term basis, and the results correlated well with 
traditional single-unit, sharp recording electrodes. As demon- 
strated by high-resolution fluoroscopy, the microwires di- 
verge a few millimeters when in position. Physical divergence 
of the microwires and independent firing characteristics 
among separate channels together suggest that the recordings 
arise from different neurons. This ability to record from 
multineuronal ensembles is a critical step in the design of a 
motor BMI neuroprosthesis, because up to 50 to 100 neurons 
may be needed simultaneously to provide sufficient predictive 
accuracy for task performance. 


Individual Neurons and the Motor Task 


Our results confirm that many neurons in human subcorti- 
cal motor areas are actively involved in hand-gripping force 
motor tasks (18, 19). Although subcortical neurons are com- 
monly analyzed as a standard aspect of movement disorder 
surgery, by assessing a subjective relationship to passive or to 
active movement, a timed behavioral task to define the spe- 
cific relationship between neuronal activity and a task is crit- 
ical (19). From our peri-event histograms, aligned with tran- 
sitions in target location, 81% of VOP/VIM neurons and 61% 
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of STN neurons showed statistically significant modulation 
during the hand-gripping force task, consistent with nonhu- 
man primate, mammalian, and human studies of motor activ- 
ity in these structures (7, 18, 19, 30). Different neurons re- 
corded from the microwire arrays clearly demonstrated a 
wide variety of task-related responses, indicating that these 
neurons are functionally independent, even though separated 
by only a few hundred micrometers within these small, sub- 
cortical structures. 

Although only a subset of neurons demonstrated a clear 
relationship to the task, this is expected, considering that only 
a fraction of neurons in either STN or thalamus respond 
directly to hand tasks, compared with either no response or a 
response to motion of other body regions. Thus, our mul- 
tichannel array results are very different from previous single- 
channel studies, in which the single-channel electrode could 
be moved and recordings could be limited to responsive neu- 
rons. In this case, the multichannel electrode array was placed 
in a fixed position and maintained at that position for several 
minutes to allow the neurons to recover before the recording 
session was started. 

All of the recording sessions were performed in patients 
undergoing movement disorder surgery for Parkinson’s dis- 
ease or for tremor disorders. Responses to the motor task were 
similar to those observed in normal, nonhuman primates. It 
therefore seems that qualitatively similar results would be 
likely to be obtained in subjects without movement disorders, 
such as quadriplegics requiring a motor BMI neuroprosthesis. 
Differences between our subjects and future BMI patients may 
be particularly pronounced in STN. Because all of the STN 
neurons were recorded in severe Parkinson’s patients not 
receiving their medications for more than 12 hours (the off- 
medication behavioral condition), it is possible that the mod- 
ulation of neuronal firing rate with hand activity could be 
different (i.e., more STN neurons responsive to the behavioral 
task) in patients receiving medication (the on-medication be- 
havioral condition), which was not tested. In addition, the on 
state may enhance the reactivity of the neurons to the task. 
Such considerations may have some bearing on the relatively 
pronounced and rapid modulatory effects observed in VOP/ 
VIM compared with STN (Fig. 3). Thus, the off state of the 
Parkinson’s disease could potentially dampen the correlation 
between neuronal firing and the behavioral task. 


Multineuronal Ensembles and Motor Task Prediction 


There are several ways to analyze the relationship between 
multineuronal ensembles and the synchronous behavioral 
task. We have chosen a predictive model, which is a stringent 
requirement but one that is critical for BMI performance. To 
achieve a high predictive accuracy, there must be multiple 
single neurons within the ensemble, which together provide 
information on the motor task structure. In addition, real-time 
predictions are critical to define motor performance for a 
realistic implementation of a human neuroprosthetic BMI. 
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The predictive algorithms used in this study interpret infor- 
mation contained within neuronal activity before and not after 
the motor activity of interest, suggesting that the neuronal 
ensembles encode a motor plan rather than merely reflecting a 
past history of sensory activation. STN and VOP/VIM neu- 
rons typically become active before muscle activation and 
movement initiation rather than afterward (7). The predictions 
(for four sessions) shown in Figure 4 are also highly robust, in 
that the neuronal ensembles reliably encode both multiple 
speeds and ranges of force. Likewise, as shown in Figure 4E, as 
model training periods are lengthened, predictions may im- 
prove. In addition, there may be improvements with increased 
numbers of neurons. The demonstration that even a few short 
sessions (up to 5 min) of multineuronal recording activity can 
show predictive accuracy for the task clearly confirms the 
feasibility of this method of analysis for the human data and 
the likelihood that these results might be extended to chronic 
recording conditions. 


Correlation with Nonhuman Primate Data 


Several groups have demonstrated the feasibility of a BMI 
neuroprosthesis in nonhuman primates, but only from cortical 
recording sites in known motor areas (primary motor, premo- 
tor, and posterior parietal). The general principles demon- 
strated by these studies include the requirement for a large 
number of neurons to fully elaborate a motor plan (a mini- 
mum of 50-100), the use of efficient models to correlate the 
neuronal firing characteristics with the behavioral task, and 
considerable animal training over days to weeks. However, 
there have been several limiting features in the translation of 
this concept to humans. First, the only FDA-approved (for 
chronic implantation) single-unit electrodes for humans have 
been the neurotrophic cone electrodes (14, 15), and these elec- 
trodes are limited for multineuronal ensembles. None of the 
electrodes used in the nonhuman primate studies are FDA- 
approved, and many of these designs are not feasible to con- 
sider for humans because of their materials (i.e., tungsten or 
silicon rather than platinum-iridium), methods of placement, 
lack of long-term ability to record neurons, and manufacture. 
In addition, none of the electronics are implantable at this 
point, particularly for a 32-channel multineuronal system. 
Many of the critical engineering and design questions remain 
unanswered for a human neuroprosthetic system, awaiting 
the clear demonstration of feasibility of the principle of corre- 
lating multineuronal ensemble outputs with a behavioral task. 

Our present data start to provide these initial feasibility data 
for short-term recordings from subcortical structures. Al- 
though all of the nonhuman primate neuroprosthetic data are 
from cortical sites, in humans it may be also feasible to con- 
sider subcortical motor sites, particularly thalamus, because of 
the considerable clinical experience with this location and the 
enhanced stability of the electrodes within the brain. In addi- 
tion, a proposed motor neuroprosthetic system may seem to 
resemble a DBS system in many ways, with a 1-mm recording 
electrode, a computational box, and radiofrequency transmis- 
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sion to a motor neuroprosthetic actuator (21, 22). Because all of 
the DBS systems are placed in subcortical targets, the direct 
clinical translation from DBS to a motor neuroprosthesis in the 
same location may be a shorter path. 

Potentially, the ability of the approach proposed here could 
be substantially improved by increasing the model training 
time to allow a better predictive accuracy, increasing the num- 
ber of neurons driving the BMI and extending the task train- 
ing time after the model training has been established (13, 22, 
24). Therefore, our results strongly suggest that subcortical 
motor areas may provide additional sites from which to derive 
direct brain control signals for human neuroprosthetics appli- 
cations. Our efforts also support the contention that many of 
the BMI principles derived from work with nonhuman pri- 
mates are highly relevant to the human intraoperative setting. 
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COMMENTS 


Pp: et al. describe their experience measuring thalamic and 
subthalamic nucleus neuronal activity during the execu- 
tion of visual motor tasks. The findings are gleaned from a 
small number of patients who underwent surgical treatment 
for Parkinson’s disease. In these patients, the authors observed 
that ensemble recordings of multiple neurons analyzed 
through a decoding algorithm correlated well with the task of 
hand gripping on cue. 

This study has several important implications for neurosur- 
gery. First, the authors effectively and ethically used the op- 
erating room to explore brain physiology. Working through 
their institutional review board, they were able to perform the 
additional testing during deep brain stimulator lead implan- 
tation without incurring additional morbidity. Although this 
is not a new paradigm, it is important because the data col- 
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lected from primate research must be challenged ultimately in 
the human model. 

The clinical relevance of this work is apparent. The complex 
organization and function of the central nervous system pre- 
cludes biological attempts at neurorestoration from being re- 
alized in the near future. However, rapid progress in the fields 
of microprocessor and microelectrode technology may offer 
solutions to disabling neurological disorders. The concept of a 
brain-machine interface (BMI), whereby conceived motor 
tasks are translated and even executed with the assistance of 
electrical-mechanical devices, soon may become reality. 


Michael Y. Wang 
Los Angeles, California 


I: this article, the authors examine the feasibility of using 
neurons in the subcortical motor areas to drive a human 
BMI. During implantation of deep brain stimulators, ensemble 
recordings were made by use of microwire arrays from neu- 
rons in either the subthalamic nucleus or thalamic motor 
areas, while patients performed a visual feedback/hand grip 
force task. The recordings then were analyzed using linear and 
nonlinear decoding algorithms to predict task activity. The 
authors demonstrate that multiple individual neurons modu- 
late firing with task performance and that excellent predic- 
tions of grip force could be obtained from measurements of 
the ensembles of neurons. This suggests that ensembles of 
subcortical neurons may be used as a source of motor control 
signals to drive a neuroprosthetic device. This article repre- 
sents the first simultaneous recording of sizable numbers of 
human subcortical neurons, and the recordings from these 
neurons can be used together to predict motor activity. Fur- 
thermore, as the authors state, this work represents an impor- 
tant bridge between nonhuman primate laboratory studies 
and the human intraoperative setting. 


Charles Y. Liu 
Los Angeles, California 


he authors describe the successful recording of multiunit 

activity from ventralis oralis posterior/ventralis interme- 
dius and subthalamic nucleus, and they correlate this with 
contralateral grip force in a brief intraoperative recording 
session. They have managed not only to record usable units; 
they also have been able to demonstrate a relationship be- 
tween the activity of the majority of neurons and grip force. 
This is impressive work, and it may open the door to thinking 
about subcortical structures as a probe for BMI, which was 
heretofore unexplored in humans. 

The authors’ implicit contention is that subcortical struc- 
tures encode movement with a degree of specificity analogous 
to the primary motor cortex. This is somewhat implausible, 
given what we know of the motor system. There is a vast 
difference between a gross measure of force and accurate and 
finely tuned movements mediated by the motor cortex. I have 
doubts as to whether this approach ever will be able to control 
useful movement from a BMI. 
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The paradigm also is an area of concern. It is clear that 
certain cells in the ventralis oralis posterior/ventralis interme- 
dius and subthalamic nucleus have kinesthetic receptive 
fields. That movements of the extremity can yield a positive 
correlation to the firing behavior of neurons within these 
structures is not surprising. This is almost certainly on the 
afferent side of what in this case is a task involving closed- 
loop motor control. How this kinesthetic feedback could per- 
tain to a BMI in a quadriplegic patient or amputee is not at all 
apparent. 


Kim J. Burchiel 
Portland, Oregon 


Ps et al. inserted multiple microwires to fan out within the 
thalamus or the subthalamic nucleus in patients undergo- 
ing functional neurosurgery for movement disorders. They 
demonstrated that these microwires can record for individual 
neurons, and in so doing, they were able to record multiple 
units simultaneously. The authors further demonstrated that 
these units are responsive, as one would expect, to motor 
inputs. In addition, they demonstrated that the activities from 
these units could be used to predict and in theory serve as the 
input station of a device that would analyze and direct the 
output to a machine. 

This is pioneering work that represents an important first 
step toward the development of a BMI. The only point of 
criticism is the authors’ suggestion that in parkinsonian pa- 
tients, the number of units or patients’ sensitivity of move- 
ments may increase in the on-medication state. This is an 
error, as it is well known that in the parkinsonian state, there 
is defocusing of motor units and an excessive number of units 
respond to movement, which is reduced with use of dopamine 
agonists such as apomorphine (1, 2). 

Another practical point is that the excursion of the 32 mi- 
crowires is 4 to 5 mm, which would take up the majority of the 
ventralis intermedius or the large part of the subthalamic 
nucleus. Could this alone produce a microlesion effect? It 
would interesting to learn whether the authors observed a 
significant benefit merely from the introduction of these 32 
wires, and how they perceive the safety of introducing such 
wires within these brain targets. 


Andres M. Lozano 
Toronto, Ontario, Canada 
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he authors describe a novel study with potential that far 
exceeds the conclusions reached in this article. Subcortical 
recordings were obtained from patients undergoing deep 
brain stimulation procedures for Parkinson’s disease. Ensem- 
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ble extracellular neuronal recordings were obtained from both 
the subthalamic nucleus and ventralis oralis posterior/ 
ventralis intermediate nucleus during a visual feedback/hand 
grip force task. The electrode data were then used to train a 
model to predict grip force on the basis of a set of recordings. 
It is notable that the data set used was the set of neuronal 
activity 1 second before movement, which therefore was rep- 
resentative of the “motor planning” stage rather than activity 
as a response to movement. Before this study, cortical record- 
ings had been used in monkeys to control a robotic arm using 
similar predictive model techniques. This article represents 
the necessary transition from nonhuman primates to humans 
so that this technology may become a clinical application. That 
subcortical targets were used in this study, rather than cortical 
recording, confers several practical advantages. The homun- 
culus representations in subcortical nuclei are more compact 
than at cortical surfaces, thus minimizing the size of electrode 
arrays needed for recording. In addition, fixation may be more 
stable with smaller subcortical implants than with larger sur- 
face grids. 

The implications of this technology extend to creating a BMI 
that could be used by patients with severe neurological com- 
promise such as spinal cord injury or stroke; however, this 
study was performed in relatively intact patients (no focal 
injury). The predictive models used herein, as well as in the 
nonhuman primate studies, require a model training session 
to acquire data using real-time recordings on the basis of 
motor responses. Individuals with neurological compromise 
will not be able to “train” the predictive models. On the basis 
of neuronal structure and electrode placement, recordings 
from individual patients are likely to be unique. Therefore, 
models cannot be trained by one person and used by another. 
It also has been demonstrated that after peripheral injury, both 
cortical and subcortical neurons reorganize and continue to 
change for some time after the acute event. Will this plasticity 
affect predictive responses in the absence of trophic factors 
stabilizing neuronal units? Regardless, this article represents a 
major step toward developing a BMI for human use. This is 
the beginning of what will be a very exciting technological 
advance in neurosurgery. 


Lee Tessler 
Patrick J. Kelly 
New York, New York 


he concept of a BMI is the subject of great science fiction 

writing. Only recently has it come to be considered a 
potential reality. Within the last decade, BMI has become an 
area of active research pursuit. Multiple and seminal concep- 
tual breakthroughs were required for the first successful dem- 
onstration of the proof of principle for a direct brain-computer 
interaction (9). This required a stable chronic recording plat- 
form, located in an area related to motor activity, which could 
be directed to change via cognitive activity in the presence of 
complete paralysis and obtain signals that could be transmit- 
ted and processed to provide useful applications. The culmi- 
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nation was a subsequent demonstration that by thought alone, 
a patient could move a cursor on a computer screen and type 
out responses to specific inquiries (10). This resulted in con- 
siderable recognition in the press and, a World Technology 
Award for Health and Medicine. More importantly, however, 
colleagues were stimulated to pursue the goal of BMI use for 
more advanced purposes such as controlling neuroprosthesis. 

This article demonstrates use of the operating room as an 
exquisite laboratory to investigate the human brain. Using the 
opportunities available during surgery for deep brain stimu- 
lation, the authors demonstrated conclusively that subcortical 
areas could be used as a source of motor control signals for a 
BMI. Their use of an electrode array to predict motor activity 
with a relatively small number of neurons for a specific task 
represents a significant advancement. That neurons could be 
functionally independent although separated by “a few hun- 
dred microns” is not new, as in these small subcortical struc- 
tures, microelectrode recordings of individual units can iden- 
tify cells that have different kinesthetic functions only tenths 
of microns apart (8, 11, 16). That there could be plasticity 
changes in only a few trials, however, is much more interest- 
ing and potentially more useful. 

The evidence for neuroprosthesis is based on identification 
of a large number of neurons, usually ranging from 50 to 100, 
that would need to be recorded simultaneously to provide 
sufficient predictive accuracy for specific task performance. 
Thus, to pick up a glass with a prosthetic device, one must 
reach out to the glass with the appropriate trajectory and 
distance, so that the glass is not knocked over, and end close 
enough to grasp the glass. In the process of grasping, one must 
then hold the glass tightly enough that it does not fall but not 
so tightly that it shatters. The return can be equally precarious, 
requiring the need for a smooth approach to the mouth by a 
different pathway so as not to undershoot or overshoot while 
carrying the weight of the glass and maintaining the fluid 
horizontal. This is an extremely demanding task that currently 
is beyond our abilities. Multineuronal assembly allows a num- 
ber of these aspects to be examined individually. Considering 
only one phase, grasping, which was the task studied in this 
article, there is a suggestion that a smaller number of cells may 
be necessary to perform it. Nonetheless, even if we were to 
identify 100 neurons that allowed accurate prediction of the 
pathway necessary to reach out to a glass, grasp it appropri- 
ately, and it bring it close enough to take a drink, this assem- 
bly still would fail in a fundamental way to bring a human 
neuroprosthesis closer to reality. 

We will not be able to identify these 100 cells and their 
functional properties before the catastrophic event that neces- 
sitates prosthesis. The damage already will have been done. It 
is necessary to identify units that will fire when reaching, 
grasping, and returning to the mouth is simply conceptualized 
although the patient is physically incapable of executing the 
movements. This brings a quite different set of needs to the 
forefront. The motor cortex is not static; it demonstrates plas- 
ticity after a variety of injuries (3, 7, 12, 15). Cortical represen- 
tation changes and results in changes in latency, potential, 
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conduction, recruitment, and so forth (1, 2, 4, 5, 13, 14). Early 
intervention with conditioning and modification of the unit 
activity might be essential to BMI success. 

It would be useful for the authors to repeat this study but 
instead of actually gripping the device, having the patients 
simply envision grasping and adjusting the force cognitively 
rather than physically to determine whether the same type of 
success can be achieved. An additional problem with classical 
neuronal conditioning is the need for sensory feedback. Con- 
trol of individual motor units can be lost during conditioning 
in the absence of peripheral sensory feedback (18). Nonethe- 
less, we have demonstrated that through a variety of auditory 
and visual sensory responses given to a patient, lack of sen- 
sory feedback responses can be overcome (9, 10). Properly 
conditioned, even individual pairs of neurons can be trained 
to co-vary in their performance, and they can be trained 
separately (6, 17, 19). The demonstration of behavioral condi- 
tioning is essential to render feasible the principle of correlat- 
ing multiple neuronal ensemble output into a behavioral task 
that can be used in human neuroprosthetics. 

The authors tend to favor the subcortical structures as a 
starting point for such a neuroprosthetic BMI. Their main 
point is that familiarity with the deep brain stimulation system 
is an excellent starting point. This argument is fallacious. 
Safety is a far more important concern, cortical approaches are 
safer, and a small cortical hemorrhage is easily identified at 
the time of surgery and usually is less serious compared with 
a subcortical hemorrhage, which may not be identifiable until 
too late. Furthermore, much of the predictive nature of cortical 
structures is already known and can be directly approached 
along the surface of the brain. The only advantage of the 
subcortical sites is that they provide an area in which a large 
number of neurons related to different functions are congre- 
gated very closely together. Thus, a simple single array could 
capture neurons involved in multiple functions of the upper 
and lower extremities. This certainly is worthy of further 
exploration. 

This is groundbreaking work. There is still a tremendous 
amount that needs to be done to bring BMI into the forefront 
in neurosurgery. I prefer to call this area of research cognitive 
engineering, as the name encompasses both the ends and the 
means of the endeavor. Deciphering how this can be accom- 
plished gives us the stimulation, motivation, and indeed the 
joy to be functional neurosurgeons. 


Roy A.E. Bakay 
Chicago, Illinois 
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Subcortical Neuronal Ensembles: An Analysis of Motor Task 
Association, Tremor, Oscillations, and Synchrony in Human 
Patients 
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Edmond and Lily Safra International Institute of Neuroscience of Natal, 59066-060 Natal, Brazil 


Deep brain stimulation (DBS) has expanded as an effective treatment for motor disorders, providing a valuable opportunity for intraop- 
erative recording of the spiking activity of subcortical neurons. The properties of these neurons and their potential utility in neuropros- 
thetic applications are not completely understood. During DBS surgeries in 25 human patients with either essential tremor or Parkinson’s 
disease, we acutely recorded the single-unit activity of 274 ventral intermediate/ventral oralis posterior motor thalamus (Vim/Vop) 
neurons and 123 subthalamic nucleus (STN) neurons. These subcortical neuronal ensembles (up to 23 neurons sampled simultaneously) 
were recorded while the patients performed a target-tracking motor task using a cursor controlled by a haptic glove. We observed that 
modulations in firing rate of a substantial number of neurons in both Vim/Vop and STN represented target onset, movement onset/ 
direction, and hand tremor. Neurons in both areas exhibited rhythmic oscillations and pairwise synchrony. Notably, all tremor- 
associated neurons exhibited synchrony within the ensemble. The data further indicate that oscillatory (likely pathological) neurons and 
behaviorally tuned neurons are not distinct but rather form overlapping sets. Whereas previous studies have reported a linear relation- 
ship between power spectra of neuronal oscillations and hand tremor, we report a nonlinear relationship suggestive of complex encoding 
schemes. Even in the presence of this pathological activity, linear models were able to extract motor parameters from ensemble dis- 
charges. Based on these findings, we propose that chronic multielectrode recordings from Vim/Vop and STN could prove useful for 
further studying, monitoring, and even treating motor disorders. 


2002). The dorsolateral STN receives afferents from motor cor- 
tex, premotor cortex, and supplementary motor areas (Parent 
and Hazrati, 1995; Hamani et al., 2004). Vim projects to these 
areas as well as receiving afferents from the ipsilateral cerebellum. 

Individual human Vim/STN neurons are active during voluntary 
and passive movement, somatosensation, and motor imagery (Lenz 


Introduction 

Neurosurgical implantation of deep brain stimulation (DBS) 
electrodes is an efficacious treatment for both Parkinson’s disease 
(PD) and essential tremor (ET) (Ondo et al., 1998; Koller et al., 
2001; Kumar et al., 2003; Rodriguez-Oroz et al., 2005; Deuschl et 
al., 2006). Common DBS targets include the subthalamic nucleus 


(STN; for PD patients) and the ventral intermediate nucleus of 
thalamus (Vim; for ET patients). Both structures are involved in 
motor control (Parent and Hazrati, 1995; Guillery and Sherman, 
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et al., 1990, 1994, 2002; Raeva et al., 1999; Magarifios-Ascone et 
al., 2000; Magnin et al., 2000; Rodriguez-Oroz et al., 2001; 
Abosch et al., 2002; Benazzouz et al., 2002; Theodosopoulos et al., 
2003; Williams et al., 2005). Many reports have examined single- 
unit activity in these regions with respect to tremor (Lenz et al., 
1988, 1994, 2002; Zirh et al., 1998; Magarifios-Ascone et al., 2000; 
Magnin et al., 2000; Rodriguez-Oroz et al., 2001; Brodkey et al., 
2004; Hua and Lenz, 2005; Amtage et al., 2008) and pathological 
synchronous oscillations (Levy et al., 2000, 2002; Amirnovin et 
al., 2004). However, the number of simultaneously recorded cells 
in these studies was low (=2), providing limited information 
about the pathological activity of larger neuronal ensembles. 
Ensembles of simultaneously recorded neurons have been 
used to enable brain-machine interfaces (BMIs) for neuropros- 
thetic control in animal models (Chapin et al., 1999; Wessberg et 
al., 2000; Nicolelis, 2001; Serruya et al., 2002; Taylor et al., 2002; 
Hochberg et al., 2006; Fetz, 2007; Patil and Turner, 2008; Nicole- 
lis and Lebedev, 2009; Lebedev et al., 2011; O'Doherty et al., 
2011). However, recordings in humans have rarely been ex- 
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tracted from neuronal ensembles (Kennedy and Bakay, 1998; 
Kennedy et al., 2000, 2004; Patil et al., 2004; Quiroga et al., 2005). 
Human subcortical ensembles have only been used ina sole study 
(Patil et al., 2004), in which our laboratory demonstrated the 
feasibility of a subcortical motor BMI using motor control signals 
from neuronal ensembles recorded in motor thalamus [Vim/ 
ventral oralis posterior motor thalamus (Vop)] or STN during 
DBS surgery to decode modulations of hand force during a one- 
dimensional target-tracking task. 

We now examine neuronal population firing patterns during a 
voluntary motor task in a new sample of 25 human patients. While 
patients performed contralateral hand movements to acquire visual 
targets, up to 23 subcortical neurons were recorded simultaneously 
and acutely in Vim/Vop and STN to elucidate the relationship be- 
tween neuronal modulations, rhythmic oscillations, and neuronal 
synchrony. A linear decoder model was applied to reconstruct cur- 
sor position from spiking activity. Neurons were classified by oscil- 
latory firing patterns, tremor association, synchrony, and tuning to 
target and movement parameters. 


Materials and Methods 


Intraoperative recordings were conducted in 25 human patients under- 
going placement of therapeutic DBS implants in either Vim or STN. All 
studies were approved by the Duke University Institutional Review 
Board and human ethics committees, and all participating patients un- 
derstood and signed all required consent forms. 

Patient characteristics and operative plan. All patients selected for this 
study underwent either Vim (N = 14, 10 male, 4 female) or STN (N = 11, 
10 male, 1 female) DBS electrode implantation surgery. Patients whose 
symptom presentation was dominated primarily by medication-resistant 
tremor (either essential tremor or severe parkinsonian tremor) were can- 
didates for implantation in Vim, while patients with severe PD (typically 
akinetic/rigid variant with on/off fluctuations and dyskinesia) were can- 
didates for implantation in STN. Both groups of patients were off their 
medications before and during surgery. 

Patients first underwent Leksell frame placement, followed by a MRI 
scan to localize the implantation target. For Vim patients, the target 
was typically estimated according to anterior—posterior commissure 
(AC-PC) criteria, located ~5—6 mm in front of the PC, on the AC-PC 
line with a lateral measure depending on the width of the third ventricle 
(typically 12-15 mm). Figure 1 A shows a typical electrode trajectory. The 
first penetration for approaching Vim was the localization pass from a 
frontal burr hole, with the upper 5 mm of the recording track near the 
border of Vop and Vim, and the lower 5 mm in Vim and close to ventralis 
caudalis at the most posterior extent. Typically, the upper 5 mm was the 
best for multineuron recordings, reflecting more of the anterior motor 
thalamus (Vop rather than Vim), as shown in Figure 1A. For STN pa- 
tients, the target was calculated by indirect methods, based on the AC-PC 
and 1 mm axial cuts of spoiled gradient recalled acquisition in the steady- 
state imaging. The initial target was located at 11-12 mm from the mid- 
line, 2-3 mm posterior to the midpoint of the AC-PC line, and 4 mm 
below the AC-PC line (Deuschl et al., 2006). Single-unit recordings were 
first performed to define the borders of the STN, according to standard 
electrophysiological criteria, with the goal of attaining at least 5.5-6 mm 
of STN. Typically, 2-3 passes were performed. Localization was per- 
formed using single-channel tungsten microelectrodes. 

For both targets, once single-unit recordings had been performed for 
localization, a 32-channel Pt/Ir microwire (35 wm diameter) array (Ad- 
Tech Medical Instrument) was passed to the appropriate depth via an 
outer cannula (Patil et al., 2004), where significant activity was noted 
with the single-unit electrode. After allowing a few minutes for initial 
recordings to stabilize, the microwire array was slowly advanced through 
the cannula. Once the number of clearly distinguishable single units was 
maximized, the microwire array was left in place. At each electrode 
depth, the patient was instructed to proceed with the voluntary motor 
task. 
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Figure 1. A, Sagittal diagram of the human brain at 15 mm off the midline. The vertical line 
represents the midpoint of the AC-PC line, which is arbitrarily set here as 24 mm in total length. 
The distance from the midpoint to the PC is 12 mm. The angled line represents a typical elec- 
trode trajectory, with the box emphasizing the best location (in terms of cell density) for the 
placement of the electrode array. This area overlaps Vim/Vop in the motor thalamus. Note that 
the DBS electrode would typically be placed deeper, at the border of Vim and ventralis caudatus 
(Vc, the sensory nucleus). The exact angle of the trajectory varied slightly from patient to 
patient, depending on the frame orientation and location of the frontal burr hole in front of the 
coronal suture. IC, Internal capsule; SNr, substantia nigra pars reticulata. B, Visualization of all 
waveforms from 20 sorted units from Patient M (Vim/Vop). Waveforms are 32 samples long and 
triggered by threshold crossing at sample 8. Individual traces were smoothed and summed into 
an accumulation buffer. Lighter colors represent a higher density of voltage traces. Traces are 
labeled by channel followed by unit number. Voltage amplitudes are normalized for display and 
are therefore represented in arbitrary units. 


Following completion of the multichannel recording sessions, the mi- 
crowire array was removed and the DBS treatment electrodes were im- 
planted. As a clinical routine, a brain computed tomography scan was 
performed within 12 h of the surgery procedure, and in no instance was 
a hemorrhage or other complication noted. Hence, the clinical risks of 
temporary placement of the 32 channel microwire array were demon- 
strated to be very low, as previously reported (Patil et al., 2004). Further- 
more, in our research, we have used this electrode in many (N = 72) 
patients over several years with no post-op evidence of hemorrhage. 

Electrophysiological recording. The 32-channel microwire recordings 
were performed with a Plexon MAP system. Since this study was per- 
formed intraoperatively during electrophysiological mapping of the im- 
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Figure 2. A, Diagram of the bidirectional hand task. The patient uses graded opening/clos- 
ing of the hand to actuate a one-dimensional cursor toward randomly appearing targets. B, 
Example off-line prediction of hand/cursor position. Prediction was obtained using a linear 
Kalman filter with a 500 s training period, from Patient M (Vim/Vop). Forty-eight units (16 
single units, 32 multiunits) were included in the ensemble. Position is given in normalized units. 


plantation sites, the recordings for each patient consisted of one or more 
epochs (up to 8, mean = 3.8), between which the electrode depths were 
altered. For each recording epoch, single units were sorted offline using 
custom software developed in-house. Extracted spikes consisted of 32 
samples each, sampled at 40 kHz, aligned on the crossing of a linear 
voltage threshold. Sorting was done by projecting these waveforms into a 
two-dimensional principal component space. Clusters in principal com- 
ponent space were determined visually and selected by use of a lasso tool 
to define spatial boundaries. If the electrodes were moved (clearly visible 
as the microdrive corrupted the recording traces) or threshold was oth- 
erwise changed during a recording epoch, the record was broken into 
separate epochs. Recorded neuronal discharges on a given channel that 
could not be sorted and isolated as single units were classified as belong- 
ing to a multiunit, indicating a collection of individual discharges whose 
identity could not be firmly established. Figure 1 B shows the sorted unit 
waveforms from a single recording session. We estimated the signal-to- 
noise ratio (SNR) by dividing the variance of the extracted spike samples 
by an estimate of the noise variance (Bankman et al., 1993), yielding a 
mean SNR of 4.69 for all sorted units. 

Voluntary motor task. Patients were placed in a supine, semisitting 
position in front of a computer monitor. A 5DT Systems Data Glove 5 
Ultra haptic glove was placed over the hand contralateral to the micro- 
electrode array. This glove was used to measure flexion/extension of the 
fingers, sampled at 1 kHz. The average flexion/extension signal, which 
effectively measured opening/closing of the hand, was used to control the 
one-dimensional position of the cursor ona video screen, placed directly 
in front of the patient for high visibility. Patients were trained to modu- 
late the opening and closing of their hand to acquire targets by moving the 
cursor into a box placed randomly along a horizontal line (Fig. 2A). 
The required target hold time was 200 ms. Once the target was acquired, 
the box disappeared for 300 ms before reappearing in a new random 
position, chosen froma uniform distribution representing the horizontal 
extent of the screen. Therefore, movements did not strictly alternate 
between left and right; successive jumps would frequently occur in the 
same direction. 

During the preliminary training/calibration phase, the cursor gain, 
offset, and target box size were systematically calibrated by the experi- 
menters to compensate for variations in physical ability. Specifically, in 
patients exhibiting limited hand mobility, the gain from hand movement 
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to cursor motion was increased. In patients with pronounced hand 
tremor, the size of the target was increased. Finally, the offset was set so 
that the resting position of the patient’s hand corresponded to the middle 
of the screen. During the recordings that followed, the length of individual 
motor task sessions varied depending on electrophysiological recording 
quality and the level of patient fatigue. Figure 2B shows a representative 
snapshot of the motor task performed by an ET patient. Note the slight 5 Hz 
tremor that occurred during target hold periods. 

Neuronal tuning to target and movement. All subsequent data analyses 
were performed using Matlab (MathWorks). We use the term “tuning” 
to refer to modulation of neuronal firing rates that is correlated to an 
external parameter and “tuning strength” to refer to the extent of those 
modulations. 

For all sorted single units and multiunits, perievent time histograms 
(PETHs) of neuronal activity (Awiszus, 1997) were generated using one 
of two event triggers: (1) the appearance of a new target or (2) movement 
time. PETHs triggered on target appearance were constructed using a 
window beginning 0.5 s before each event trigger and ending 1.5 s after, 
whereas a symmetric 2 s window was used for PETHs triggered on move- 
ment time. Movement time was defined as the moment at which the 
cursor crossed the midpoint between initial cursor position (at target 
appearance) and the endpoint target position. We chose this standard as 
a robust definition of movement execution in light of patient tremor and 
occasional incorrect movements. Regardless of reaction time, this event 
trigger was locked to movement, being in close proximity to the point of 
maximum hand velocity before target acquisition. Trials with anomalous 
movement times <200 ms (premature movement) or >1000 ms (inat- 
tention) were discarded. Only neurons with at least 50 valid target acqui- 
sition trials were chosen for further analysis. 

Neuronal tuning to either target or movement was determined by 
quantifying the deviation of each PETH from a bootstrap distribution of 
PETHs generated by uncorrelated triggers. We calculated significance 
using the one-sample Kuiper’s test (Kuiper, 1962; Batschelet, 1981; Zar, 
1999), a nonparametric test related to the Kolmogorov—Smirnov (K-S) 
test (Zar, 1999) but better suited for nonbiased PETH analysis. Unlike the 
K-S test, Kuiper’s test is equally sensitive throughout the distribution, a 
useful property in scenarios in which the locations of the peak modula- 
tions are not known a priori. Variations of the K-S test have been used 
previously in the significance evaluation of neuronal PETHs (Ghazanfar 
et al., 2001; Wiest et al., 2005; Gutierrez et al., 2006). In this study, we 
used Kuiper’s test to distinguish an observed distribution of event- 
triggered spike times from the null hypothesis (uniform probability dis- 
tribution). Kuiper’s test requires the calculation of the maximum 
positive and negative deviations of the observed PETH cumulative 
distribution function (CDF) from a uniform distribution CDF (ramp 
function); the sum of these two deviations is the statistic V: V = 
max[CDF,ampie ~ CDF uniform) + max[CDF, — CDF, amplel+ 

The Kuiper statistic, K, is a normalized version of V, taking into ac- 
count the size of the observed sample size N, in this case, the number of 
binned spikes: (K VN? + 0.155 + 0.24N~'/?). To distinguish 
the test statistic K,,, from the null hypothesis, we generated a boot- 
strapped distribution of 1000 simulated Kuiper statistics (K,;,,,). Prelim- 
inary analysis determined shuffling of spike timestamps to be a 
suboptimal control; the process eliminates spike autocorrelations from 
the bootstrap distribution, thereby potentially biasing the evaluation of 
the observed distribution in favor of significance. Instead, each value of 
Kim was calculated using a PETH constructed from the original spike 
timestamps but processed using a distribution of randomized event trig- 
gers; the triggers were drawn uniformly from the time span of the record- 
ing session. In other words, in each bootstrapped trial, the timestamps of 
the events (target appearance, movement) are randomly and indepen- 
dently assigned to decorrelate them from the spiking data. For each 
sorted unit, the resulting bootstrapped distribution of K,;,, was used to 
produce a p value: p = (number of trials for which K%.,,, > Kp5)/(N + 1). 

Units were deemed to be tuned to task events (target or movement 
onset) using the threshold p < 0.05. These units exhibited temporal 
modulations in firing rate relative to newly appearing targets and/or 
target-directed movements. Tuning strength was defined as the z-score 
of the observed PETH relative to the bootstrap distribution. 


uniform 
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Directional tuning. The directional tuning of each sorted unit or mul- 
tiunit was defined as the difference in neuronal response for leftward 
versus rightward movements. Significance of directional tuning was deter- 
mined using the two-sample Kuiper’s test. This test applied the calculation of 
the maximum positive and negative deviations between the spike CDFs for 
leftward and rightward movements: V = max[CDF CDF ampte 2] + 
max[CDF anpie a CDF, ample il 

The Kuiper statistic K was calculated from V using the same equation 
as the one-sample Kuiper’s test, but in the case of the two-sample 
Kuiper’s test, the effective sample size (N.4) replaced N to account for the 
combined contributions of the individual sample sizes N, and Nj: Neg = 
(N,N3)/(N, + N,). 

Two PETHs triggered on movement time were generated, one for 
leftward movements and one for rightward movements. As with the 
one-sample Kuiper’s test, each unit’s K,,, statistic was compared with a 
bootstrapped distribution of K,;,, generated from randomized trigger 
times. Units were deemed to be directionally tuned using the threshold 
p < 0.05. Tuning strength was defined as the z-score of the observed 
PETHs relative to the bootstrap distribution. 

Tremor sensitivity. Although concurrent surface EMG recordings were 
not permitted under our approved experimental protocol, haptic/posi- 
tion tracking have been used repeatedly to analyze tremor (Bardorfer et 
al., 2001; Su et al., 2003; Vinjamuri et al., 2009), as has accelerometry 
(Ghika et al., 1993; Grimaldi et al., 2007; Birdno et al., 2008). 

For all sorted single units, perievent phase histograms (PEPHs) of 
neuronal activity were generated using phase of the patient’s tremor as an 
event trigger, similar to the approach of Lebedev et al. (1994). Tremor 
was determined from hand velocity, and tremor periods within the range 
of 100-2000 ms (0.5-10 Hz) were analyzed. To exclude the impact of 
voluntary movements, hand velocity peaks occurring within 250 ms of a 
movement trigger were excluded. Each tremor period was defined in 
units of phase, with neuronal spike activity captured into 100 bins of 
equivalent phase aperture (3.6° each). The zero phase for each cycle was 
defined by a local maximum in hand velocity. Only sorted units with at 
least 500 valid tremor periods were chosen for further analysis. 

Each resulting PEPH was a measurement of neuronal firing rate with 
respect to tremor phase. The one-sample Kuiper’s test, in addition to 
possessing uniform sensitivity, is also rotationally invariant, meaning 
that the arbitrary choice of zero phase has no effect on the assessment of 
statistical significance. For analysis of tremor tuning, we generated a 
bootstrapped distribution of 1000 simulated Kuiper statistics (K, 


sample 1 _ 


sima)} 
each was calculated using a PEPH constructed from trials whose binned 
spike counts were circularly rotated by uniformly random phase offsets. 
For each analyzed unit, the bootstrapped distribution was used to pro- 
duce a p value. Units were deemed to be tremor associated (tuned) using 
the threshold p < 0.05. Tuning strength was defined as the z-score of the 
observed PEPH relative to the bootstrap distribution. 

Oscillatory neurons. For all sorted units with at least 1000 extracted 
spikes, we used Welch’s method (Oppenheim and Schafer, 1975) with 
eight nonoverlapping segments to determine the spike train autopower 
spectral density. The power spectra were smoothed using a 0.5 Hz rect- 
angular sliding window. For each unit, the peak autopower frequency 
was determined in the 1-25 Hz range, with frequency content <1 Hz 
discarded for the remainder of the analysis. For the peak frequency, we 
determined the SNR by dividing peak power by the mean power (as- 
sessed from 1 Hz up to the Nyquist frequency of 500 Hz). For the purpose 
of comparison, the same spectral analysis was performed on hand accel- 
eration traces for all recorded sessions. 

Preliminary analysis indicated a functional separation of peak fre- 
quencies at ~2.5 Hz. Units with a peak power frequency <2.5 Hz tended 
to be dominated by low-frequency power and were therefore judged not 
to be sufficiently oscillatory in a physiologically relevant frequency range. 
As in previous studies (Lenz et al., 1988; Amtage et al., 2008), only units 
with a peak SNR >2 were classified as oscillatory. Sharpness of the peaks 
in either the spike train or hand acceleration autopower spectra was 
determined by calculating the maximum power concentrated in a 1 Hz 
band within the physiologically relevant 2.5-7.5 Hz window. “Peaked- 
ness” was defined as the ratio of the power in this band relative to the total 
power in the 1-25 Hz band. 
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In addition to established linear methods, we developed a heterodyne 
method for detecting nonstationary or wide-bandwidth synchronized 
activity between each neuron’s firing rate and associated hand velocity. 
This method, inspired by the frequency shifting scheme used in radio 
frequency transceivers, was applied to all sorted units that fulfilled the 
selection criteria for both target tuning analysis and oscillatory analysis. 
The spike train and hand velocity recordings were first bandpass filtered 
(fourth-order Butterworth, zero phase method) between 2 and 12 Hz, 
leaving both signals with negligible 0 Hz (DC) energy. The two signals were 
then multiplied, yielding a third time series from which to extract spectral 
energy. Because multiplication in the time domain is equivalent to convolu- 
tion in the frequency domain (and vice versa), any synchronous frequency- 
modulated components in both neuronal firing rate and hand velocity are 
transferred to DC. The ratio of spectral energy (E,,,) from 0 to 0.125 Hz 
(signal) over that from 0.25—2 Hz (baseline) was considered as a metric of 
heterodyne correlation between neuronal activity and hand movement. 
To provide a control for this estimate, a bootstrap distribution (E,;,,,) was 
generated by shuffling spike timestamps 1000 times and repeating the 
above analysis. Since all low-frequency information is filtered out before 
multiplication, shuffling timestamps was determined to be a suitable 
control. For all single units, the resulting bootstrapped distribution was 
used to produce a p value. Units were deemed to be heterodyne-tuned to 
tremor if the heterodyne correlation between neuronal activity and 
movement was significant using the threshold p < 0.05. Heterodyne 
tremor tuning strength was defined as the z-score of the observed spectra 
relative to the bootstrap distribution. 

Efficacy of neuronal recordings for kinematic predictions. Prediction al- 
gorithms from the BMI literature were applied to subcortical neuronal 
populations to extract behavioral parameters. Several algorithms were 
tested, including the linear Kalman filter, unscented Kalman filter, and 
the Wiener filter. Figure 2 B shows an example off-line prediction for a 
30 s window of task performance. 

Since all three algorithms achieved the same approximate fidelity in 
preliminary testing, we chose the Wiener filter for further analysis due to 
its computational simplicity and extensive presence in the BMI literature 
(Wessberg et al., 2000; Carmena et al., 2003; Patil et al., 2004). Individual 
Wiener filters were fit by binning neuronal data into 100 ms time slices 
with 10 causal lags and regressing against recorded hand position. Model 
training was performed by the random selection of 50% of these time 
slices; predictions were then made on a distinct random 25%. This pro- 
cess was repeated with 100 draws of fit and predict time slices. Correla- 
tion coefficient (R) between predicted hand position and actual hand 
position was measured for each of the draws; the mean correlation coef- 
ficient was reported for each recording session. Offline prediction results 
are reported for all sessions with at least 50 presented targets. 

We generated neuron dropping curves for selected sessions (Wessberg 
et al., 2000) by drawing random subsets from the neuronal ensemble. For 
each subset ensemble size N, we performed 1000 draws of random en- 
semble subset and Wiener filter fit and prediction; the R values for these 
draws were averaged to form a smooth neuron dropping curve. Follow- 
ing Wessberg et al. (2000), the resulting curve was then fit to the follow- 
ing hyperbolic function to extrapolate the performance results to larger 
ensemble sizes: R* = cN/(1 + cN). 

Neuronal synchrony. The use of simultaneous ensemble recordings 
allows for the analysis of pairwise synchrony between neurons. To deter- 
mine the statistical significance of the synchrony between two neurons, 
we analyzed the cross-correlation peak between pairs of spike trains. 
Pairs were analyzed if they each contained at least 100 spikes and corre- 
sponded to a session with at least 50 targets. The cross-correlation coef- 
ficient was first calculated for the observed spike trains of the two 
neurons, then smoothed using a 5 ms rectangular sliding window. The 
observed test statistic C,,,, was defined as the peak coefficient in the +10 
ms time lag range. Bootstrap simulations (n = 1000) of the two spike 
trains were generated by convolving the spike trains with a Gaussian 
kernel (o = 250 ms) and then generating new spike trains via an inho- 
mogeneous Poisson process. The smoothing filter was used to extinguish 
correlated high-frequency content in the bootstrap distribution while 
maintaining low-frequency correlation in mean firing rate. Each of these 
bootstrap simulations was used to produce a cross-correlation coefficient 
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Table 1. Pairwise classifications for single units 
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Target Movement Direction Tremor Oscillatory Heterodyne 
Vim/Vop cells 
Target 29.2% (N = 168) 32.0%(-+)** (N = 75) 16.0% (N = 75) 9,5%(-+)' (N = 126) 5.1% (N = 158) 3.8% (N = 158) 
Movement = 34.7% (N = 75) 14.7%(+)" (N = 75) 13.8% (N = 65) 1.4%(—)* (N = 74) 4.1% (N = 74) 
Direction —_ = 25.3% (N = 75) 12.3%(+)" (N = 65) 4.1% (N = 74) 8.1%(-+)! (N = 74) 
Tremor —_ _ _ 12.4% (N = 169) 2.4% (N = 169) 6.3%(+)* (N = 126) 
Oscillatory —_— — —_ — 21.5% (N = 274) 3.8% (N = 158) 
Heterodyne —_— - —_ — —_ 13.3% (N = 158) 
STN cells 
Target 22.9% (N = 83) 38.5%(+)** (N = 26) 11.5% (N = 26) 4.1% (N = 74) 4.9% (N = 81) 2.5% (N = 81) 
Movement —_ 42.3% (N = 26) 11.5% (N = 26) 12.5% (N = 24) 8.3% (N = 24) 8.3% (N = 24) 
Direction —_ = 19.2% (N = 26) 0.0% (N = 24) 4.2% (N = 24) 0.0% (N = 24) 
Tremor —_ —_ —_ 15.9% (N = 82) 1.2% (N = 82) 4.1% (N = 74) 
Oscillatory —_— — —_— — 17.9% (N = 123) 6.2% (N = 81) 
Heterodyne —_— — —_— — 17.3% (N = 81) 


N, Number of analyzed cells per entry. Off-diagonal table entries have a smaller number of analyzed cells because they must fulfill the criteria of two independent classification tests. (++, —), Significantly (high, low) level of pairwise 


classification (Two-tailed Fisher's exact test: ‘b <0.05, *p < 0.01, **p < 0.001). 


C.im- The bootstrapped distribution was used to produce a p value, and 
neuron pairs were deemed to be significantly synchronous using the 
threshold p < 0.05. 

To visualize the time dependence of pairwise synchrony, we generated 
joint peristimulus time histograms (JPSTHs) for neuron pairs, as origi- 
nally proposed by Aertsen et al. (1989). Our JPSTHs were adjusted by 
subtracting the shift predictor histogram and normalizing (bin-by-bin) 
by the standard deviation, a procedure referred to by Aertsen et al. (1989) 
as the “true normalization” of the JPSTH. 


Results 


A total of 25 DBS implantation patients were examined. In these 
patients, we simultaneously recorded from ensembles of up to 23 
well isolated neurons from either Vim/Vop or STN, depending on 
the site of electrode location. Recording sessions varied substantially 
in terms of duration and target acquisition rate, as limited by indi- 
vidual patient pathology and motivation. Neurons from these sub- 
cortical areas were classified by oscillatory firing patterns and tuning 
to target, movement, direction, and tremor. Moreover, neuronal 
ensemble data served as input for an offline linear prediction model 
to reconstruct cursor position. Finally, neuronal pairs were analyzed 
for evidence of functional synchrony. 

STN cells (N = 168) exhibited a higher (p < 0.01, Mann— 
Whitney U test) mean firing rate than Vim/Vop cells (N = 83): 
15.8 + 1.95 Hzand 11.7 + 1.02 Hz, respectively (mean + 1 SEin 
both cases). In both subcortical areas, we found substantial pop- 
ulations of oscillatory neurons, as well as neurons strongly tuned 
to target, movement, direction, and tremor. Furthermore, neu- 
rons in both subcortical areas tended to show tuning to multiple 
parameters (Table 1) rather than belonging to disjoint sets. For 
example, the number of Vim/Vop cells tuned to both target and 
tremor was higher than would be expected under statistical inde- 
pendence (two-tailed Fisher’s exact test, p < 0.05). At the ensem- 
ble level, a substantial number of analyzed cell pairs were found to 
exhibit synchrony. Curiously, all tremor-associated neurons ex- 
hibited synchrony within the recorded neuronal ensemble. 


Neuronal tuning to target and movement 

Both Vim/Vop and STN neurons represented target appearance and 
movement onset (Table 2). Of all single units tested, 29.2% of 168 
Vim/Vop cells and 22.9% of 83 STN cells were found to be tuned to 
target appearance. Both of these percentages represent signif- 
icant populations (binomial test, p < 0.001 in both cases). 
Figure 3A shows example PETHs for three highly responsive 
neurons. 


Table 2. Behavioral tuning of subcortical neurons 


Parameter Unit type Area No. of units #Tuned 
Target Single Vim/Vop 168 49** (29.2%) 
Target Single STN 83 19** (22.9%) 
Target Multi Both 753 87** (11.6%) 
Movement Single Vim/Vop 75 26** (34.7%) 
Movement Single STN 26 11** (42.3%) 
Movement Multi Both 414 48** (11.6%) 
Direction Single Vim/Vop 75 19** (25.3%) 
Direction Single STN 26 5* (19.2%) 
Direction Multi Both 414 36* (8.7%) 


Tuned population tested for significance (Binomial test: 'p < 0.05, *p < 0.01, **p < 0.001). 


As explained above, trials with reaction times outside the 200— 
1000 ms range were discarded for movement tuning, and only 
sessions with at least 50 valid trials were subjected to further 
statistical analysis. Because of the additional reaction-time crite- 
rion, fewer single units were analyzed for movement tuning than 
for target tuning. Of these, 34.7% of 75 Vim/Vop cells and 42.3% 
of 26 STN cells were found to be tuned to movement. Both of 
these percentages represent statistically significant populations 
(binomial test, p < 0.001 in both cases). Figure 3B shows exam- 
ple PETHs for three highly responsive neurons. 

We also found a strong positive correlation between the strength 
of target appearance tuning and that of movement tuning, for both 
Vim/Vop and STN cells. When controlling for the number of session 
trials, target tuning strength significantly predicted movement tun- 
ing strength (B = 0.70, p < 0.001 for Vim/Vop; B = 0.71, p< 
0.001 for STN). This result is consistent with Table 1, which indicates 
that a larger-than-expected number of neurons in both subcortical 
areas were tuned to both target and movement. 

A portion of the correlation between target tuning and move- 
ment tuning may be explained by a tight temporal offset between 
target appearance and movement time. However, visual inspec- 
tion of some tuned units indicated a clear decoupling of the neu- 
ral encoding of target appearance and movement. Sorted raster 
plots from example neurons are shown in Figure 4, A and C. 
From these, we derived the color maps in Figure 4, B and D, each 
showing two clear bands of increased spike density. In both pan- 
els, the vertical bands are independent of movement time and are 
clearly related to target appearance (~450 ms postappearance). 
The diagonal bands have a near-unity slope, indicating a clear 
time-locked relationship between neuronal activity and move- 
ment time. For both units, the second peak in firing rate occurred 
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Figure 3. Example PETHs. A, Strongly tuned units to target appearance. i, Vim/Vop cell, 
Patient M, 465 trials. ii, Vim/Vop cell, Patient M, 375 trials. dif, STN cell, Patient H, 310 trials. B, 
Strongly tuned units to movement time. i, Vim/Vop cell, Patient M, 390 trials. ii, Vim/Vop cell, 
Patient M, 310 trials. iif, STN cell, Patient H, 201 trials. C, Strongly direction tuned units. i, STN 
cell, Patient H, 201 trials. fi, Vim/Vop cell, Patient M, 416 trials. iif, Vim/Vop cell, Patient M, 394 
trials. For A, reported time is relative to target appearance. For B and C, reported time is relative 
to movement time. All plots were smoothed using a Gaussian kernel, o = 40 ms. 


~300 ms after the defined movement time. From these data, it 
can be concluded that these neurons were tuned to both target 
appearance and movement; they modulated their firing rates in 
relation to both events. 

Modest differences were seen in the aggregate response patterns 
of Vim/Vop and STN cells classified as responsive to either target or 
movement (Fig. 5). Both cell types exhibited a mean response that 
peaked following target appearance (Fig. 5A); STN cells peaked later 
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Figure 4. Separation between single-unit response to target appearance and movement 
time. Time along the x-axis is relative to target appearance. A and C show spike raster plots 
relative to target appearance, with individual trials sorted by movement time (red circles). B 
and D show smoothed color plots for the same two units as A and C, using the relative timing of 
all spikes and the movement time of their corresponding trials. The dashed line (unity slope) 
depicts movement time. For generation of the color plots, the data were smoothed using a 
two-dimensional Gaussian kernel, with o = 20 ms along the x-axis and o = 40 ms along the 
y-axis. Both units are Vim/Vop cells from Patient M. 
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Figure 5. A, B, Mean normalized PETHs for all responsive units (p < 0.05) from both sub- 
cortical areas using two event triggers: target appearance (A), and movement time (B). Re- 
ported time is relative to the event trigger. Before aggregation, individual PETHs were 
convolved with a Gaussian kernel (o = 40 ms) and normalized relative to mean firing rate. For 
both panels, the differences between the mean Vim/Vop and STN responses were statistically 
significant (y test, p < 0.001). 


on average. The mean response of the Vim/Vop cells peaked imme- 
diately before movement while that of the STN cells peaked concur- 
rently with movement (Fig. 5B). Differences in the relative lags for 
Vim/Vop and STN neuronal activation likely reflect the position of 
thalamic and STN neurons in the network hierarchy of motor con- 
trol (Marsden et al., 2001; Guillery and Sherman, 2002; Gradinaru et 
al., 2009). The motor regions of the thalamus are more involved with 
intention, with signals arriving before motor cortex activation, 
whereas the collaterals from motor cortex to STN deliver signals at 
the time of motor activation. 

For both aggregates, the differences between the mean Vim/Vop 
and STN responses were statistically significant (x test, p< 0.001). 
However, similar proportions of Vim/Vop and STN cells were tuned 
to target appearance; the same was also true for movement tuning 
(two-tailed Fisher’s exact test, p > 0.05 in both cases). 
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Table 3. Tremor tuning of subcortical neurons 


Area Unit type No. of units No. of tuned 
Vim/Vop Single 169 21** (12.4%) 
STN Single 82 13** (15.9%) 


Tuned population tested for significance (Binomial test: tp < 0.05, *p < 0.01, **p < 0.001). 


Directional tuning 

Another metric of interest for the behavioral responsiveness of 
subcortical neurons was directional tuning; Table 2 gives the di- 
rectional tuning results for both single units and multiunits. Of 
all tested single units, 25.3% of 75 Vim/Vop cells and 19.2% of 26 
STN cells were found to exhibit directional tuning. Both of these 
percentages represent statistically significant populations (bino- 
mial test, p < 0.001 for Vim/Vop, p < 0.01 for STN). Figure 3C 
shows example PETHs for three strongly tuned neurons. Similar 
proportions of Vim/Vop and STN cells were tuned to direction 
(two-tailed Fisher’s exact test, p > 0.05). 

Despite the clear separation in the neuronal response to left- 
ward and rightward movements, note the transient regions of 
convergence that occurred in Figure 3C. In Figure 3Ciii, for ex- 
ample, the neuronal responses to each direction converged just 
before movement. For many tuned neurons in both Vim/Vop 
and STN, the degree of directional modulation varied through- 
out the temporal window. 

For both Vim/Vop and STN cells, we found strong positive 
correlations between directional tuning strength and the strength 
of both target tuning and movement tuning. When controlling 
for the number of session trials, target tuning strength signifi- 
cantly predicted directional tuning strength (B = 0.16, p < 0.05 
for Vim/Vop; B = 0.38, p < 0.01 for STN). Similarly, movement 
tuning strength significantly predicted directional tuning 
strength (B = 0.24, p < 0.01 for Vim/Vop; B = 0.43, p < 0.05 for 
STN). The latter finding is consistent with the Vim/Vop pairwise 
classification result in Table 1. 


Properties of multiunits 

Whereas single units are identifiable as distinct neurons, a mul- 
tiunit is likely comprised of distant neurons with lower SNR spike 
profiles. From Table 2, it can be seen that a significant population 
of analyzed multiunits were tuned to target, movement, and di- 
rection (binomial test, p < 0.01 in all cases). A substantial num- 
ber of these tuned multiunits were found on the same recorded 
channel as tuned sorted units. Furthermore, when controlling for 
the number of session trials, the target tuning strength of single 
units significantly predicted the target tuning strength of same- 
channel multiunits (B = 0.18, p < 0.01). This confirms the pres- 
ence of correlated tuning in nearby neurons. Thus, a substantial 
amount of encoded information was present in subcortical mul- 
tiunits, arguing for the potential inclusion of these signals in fu- 
ture analyses of ensemble activity. 


Tremor sensitivity 

To identify potentially pathological neurons within the recorded 
subcortical populations, we analyzed the tremor sensitivity of 
single units using the discussed PEPH approach; the results are 
given in Table 3. Ofall single units tested, 12.4% of 169 Vim/Vop 
cells and 15.9% of 82 STN cells were found to be correlated to 
observable hand tremor. Both of these percentages represent sta- 
tistically significant populations (binomial test, p < 0.001 in both 
cases). Figure 6 shows example PEPHs for three strongly tremor- 
sensitive neurons. These results demonstrate that for highly 
tuned units, the dependence of spike rate on tremor phase re- 
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Figure6. Example PEPHs triggered on hand tremor phase for strongly tremor tuned units. A, 
Vim/Vop cell, Patient M, 4478 tremor periods. B, Vim/Vop cell, Patient M, 1259 tremor periods. 
C Vim/Vop cell, Patient M, 4642 tremor periods. Zero phase is aligned to local peaks in hand 
velocity. All panels show individual PEPHs for each temporal quarter of their respective sessions, 
overlaid with the aggregate PEPH for the session as a whole (ALL). All plots in this figure are 
smoothed using a Gaussian kernel, 0 = 15°. 


mained stable throughout the recording session (Fig. 6), even if 
the mean firing rate varied substantially Figure 6, A and C. Similar 
proportions of Vim/Vop and STN cells were tuned to tremor 
(two-tailed Fisher’s exact test, p > 0.05). 

For Vim/Vop cells (but not STN cells), we found a positive 
correlation between the strength of tremor tuning and that of 
directional tuning. When controlling for the number of session 
trials, directional tuning strength significantly predicted tremor 
tuning (6 = 0.34, p < 0.05). However, we found no relationship 
between tremor tuning and either undirected target or move- 
ment tuning (p > 0.05 for all cases). 

However, these results do not distinguish whether these 
tremor-tuned neurons are involved in a pathological mechanism 
that causes tremor or merely reflect somatosensory signals indic- 
ative of tremor. 


Oscillatory behavior 

To explore neuronal oscillations in Vim/Vop and STN and their 
relationship to patient pathology, we inspected the autopower 
spectra of single-unit spike trains for strong frequency peaks, 
yielding peak frequency and SNR (Fig. 7A). Of all tested single 
units with SNR > 2, the distribution of peak frequencies showed 
a clear bimodal distribution with a border between low- and 
high-frequency oscillations at ~2.5 Hz (Fig. 7B). 

Spike train spectra from Vim/Vop and STN neurons also 
tended to possess large amounts of energy at low frequencies, 
suggestive of 1/f (pink) noise. This power-law distribution has 
been described for cortical neurons as a stochastic process (Davidsen 
and Schuster, 2002), but may also be related to slow modulations of 
patient attention and arousal. The observed distribution may ex- 
plain the 2.5 Hz trough (Fig. 7) that separates neurons dominated by 
1/f noise from those exhibiting strong oscillations in the tremor- 
relevant frequency range (2.5—7.5 Hz). Only neurons with sufficient 
power within this frequency range were eligible to be classified as 
oscillatory. 

The oscillatory classification results are shown in Table 4; 21.5% 
of 274 Vim/Vop cells and 17.9% of 123 STN cells were classified as 
oscillatory. No difference was seen in the proportions of oscillatory 
Vim/Vop and STN cells (two-tailed Fisher’s exact test, p > 0.05). 
Figure 8 shows example interspike interval (ISI) plots for three 
highly oscillatory cells. Note that all three ISI histograms exhibit 
some degree of bimodality, indicative of periodic bursting behavior. 

Figure 9A shows the smoothed autopower spectra of spike 
trains for all analyzed single units, with each individually normal- 
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Figure 7. A, Distribution of peak frequencies for the spike train autopower spectra of 397 
analyzed single units. Units with peak frequency >10 Hz (17.4% of all units) are not 
shown in this figure; they are distributed with near uniformity in the 10 —25 Hz range with 
SNR mostly below the classification threshold of 2. B, Histogram showing the number of 
oscillatory (SNR > 2) units at each peak frequency. A clear separation between two 
subpopulations is clear at ~2.5 Hz. 


Table 4. Oscillatory Units 


Area Unit type No. of units No.of oscillatory 
Vim/Vop Single 274 59 (21.5%) 
STN Single 123 22 (17.9%) 
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Figure 8. Example |S! histograms for highly oscillatory neurons. A, Vim/Vop cell, Patient W. 
B, STN cell, Patient V. C, Vim/Vop cell, Patient J. 


ized horizontal trace corresponding to a distinct unit. From this 
figure, one can visually identify some of the highly oscillatory 
units as well as observe the congruity between multiple units 
from the same patient. The difference between the mean normal- 
ized spectra for Vim/Vop and STN cells (Fig. 9B) is statistically 
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Figure 9. A, Smoothed autopower of spike trains for all sorted units with sufficient spike 
count. The autopower spectra (determined by Welch’s method) of each horizontal trace, corre- 
sponding to a distinct unit, has been individually smoothed (using a 0.5 Hz sliding window) and 
normalized. The horizontal traces are grouped by patient and recording area. The color bar 
represents arbitrary units of normalized energy density. B, Mean autopower spectra from all 
analyzed Vim/Vop and STN cells. Spectra are individually smoothed using a 0.5 Hz sliding win- 
dow and normalized before aggregation. Each of the dashed traces represents a bootstrap 
simulation in which only half of all analyzed units are aggregated. The difference between the 
mean Vim/Vop and STN spectra is statistically significant (° test, p < 0.001). 


significant ( x? test, p <<. 0.001). From Figure 9B, it is clear that 
STN cells tended to concentrate power at a lower frequency (3 
rather than 4 Hz). 

The pairwise classification results in Table 1 reject the notion 
that oscillatory neurons and behaviorally tuned neurons form 
disjoint sets. Furthermore, we found no relationship between 
spike autopower peakedness and the strength of any of the three 
(target, movement, direction) behavioral tuning metrics (p > 
0.05 for all cases, for both Vim/Vop and STN). The lack of a clear 
anticorrelation suggests that the sets of behavioral neurons and 
oscillatory neurons are far from disjointed. Instead, they appear 
to exist as overlapping populations. 

Our next analysis intended to uncover a relationship between 
strong oscillatory neuronal patterns and observable hand tremor. 
However, we did not find any clear relationship. Linear regres- 
sion analysis revealed no relationship between peak frequency 
(2.5-7.5 Hz range) of spike train autopower spectra and corre- 
sponding hand acceleration autopower spectra (p > 0.05 for 
both Vim/Vop and STN). No relationship was found between the 
sharpness of the two spectra for Vim/Vop neurons (p > 0.05), 
but we did observe a marginally significant positive relationship 
for STN neurons (B = 0.72, p = 0.038). Figure 10 shows overlaid 
spectra for the spike train autopower and hand acceleration au- 
topower of three highly oscillatory units. For all three cells (rep- 
resentative of the population as a whole), the peak frequencies do 
not coincide. However, we did find a marginally significant (B = 
0.24, p = 0.055) correlation between spike autopower peakedness 
and tremor tuning strength for Vim/Vop cells (p > 0.1 for STN 
cells). These findings call into question the presumed causal lin- 
ear relationship between the two, suggesting the possibility of an 
elusive nonlinear relationship. 
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Figure 10. Comparison of spike train autopower and hand acceleration autopower for three 
highly oscillatory units. A, Vim/Vop cell, Patient W. B, STN cell, Patient V. C, Vim/Vop cell, 
Patient J. Note that for all three cells, peak frequency does not coincide. 


Table 5. Heterodyne tremor tuning 


Area Unit type No. of units No. of tuned 
Vim/Vop Single 158 21** (13.3%) 
STN Single 81 14** (17.3%) 


Tuned population tested for significance (Binomial test: ‘p < 0.05, *p < 0.01, **p < 0.001). 


Our heterodyne decoding analysis further explored this rela- 
tionship by applying a nonlinear frequency shifting approach to 
the autopower spectra. Of 239 analyzed single units, 13.3% of 
Vim/Vop cells and 17.3% of STN cells were found to be tremor 
associated via heterodyne decoding (Table 5). Both of these per- 
centages represent statistically significant populations (binomial 
test, p < 0.001), but the difference between them is not signifi- 
cant (two-tailed Fisher’s exact test, p > 0.05). 

Heterodyne decoding is a more sensitive method for detecting 
tremor correlations than spectral peak analysis if the tremor sig- 
nal is wide-bandwidth or prone to phase changes. Indeed, this 
schema may better serve to explain the relationship between the 
oscillatory activity of neurons and observed tremor. In fact, this 
can explain the similarity in the proportions of tuned neurons in 
Tables 3 and 5. Furthermore, Vim/Vop firing indicated a strong 
positive correlation between tremor tuning strength, identified 
using PEPHs, and heterodyne tremor tuning strength (6 = 0.31, 
p <.0.001). This relationship was marginally significant in STN 
cells (B = 0.28, p = 0.086). These findings are consistent with the 
pairwise classification results in Table 1, which indicated a higher 
than expected joint classification for the two tremor tuning anal- 
yses for Vim/Vop cells. 


Efficacy of neuronal recordings for kinematic predictions 

We also performed off-line predictions of cursor motion using 
the recorded ensembles. The correlation coefficient (mean + 
1.98 SE) for each of the sessions is shown in Figure 11. Although 
the predictions varied greatly across sessions and patients, the 
results compared favorably with our previous study (Patil et al., 
2004). The best session for each subcortical area (Vim/Vop, STN) 
was chosen for further analysis, and neuron dropping curves 
were generated for these two sessions and fitted to a hyperbolic 
function (Wessberg et al., 2000). Extrapolation of the hyperbolic 
fit produced estimates of the approximate ensemble sizes re- 
quired to achieve R* = 0.9: 106 Vim/Vop neurons or 397 STN 
neurons. 
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Figure11. Dependence of off-line BMI predictions on neuron ensemble size. Each data point 
corresponds to a recording session. Correlation coefficient (R) is indicated as mean + 1.98 SE. 
The best session for each subcortical area (Vim/Vop, STN) was chosen for further analysis. 
Neuron dropping curves are shown for each of these sessions and fitted to a hyperbolic function 
(Wessherg et al., 2000). Extrapolation calculations are presented in Results. 


Table 6. Pairwise neuronal synchrony 


Area No. of pairs No. of synchronous 
Vim/Vop 1648 708** (43.0%) 
STN 693 179** (25.8%) 


Tuned population tested for significance (Binomial test: 'p < 0.05, *p < 0.01, **p < 0.001). 


Neuronal synchrony 

We analyzed neuronal synchrony in pairs of sorted units and 
investigated how its prevalence varied across subcortical areas; 
the results are given in Table 6. Using the cross-correlation ap- 
proach, 43.0% of Vim/Vop pairs and 25.8% of STN pairs were 
found to be significantly synchronous. Both of these percentages 
represent significant populations (binomial test, p << 0.001 for 
both cases). Figure 12A shows example cross-correlation plots 
for three highly synchronous pairs, while Figure 12 B shows the 
normalized JPSTH for the same three pairs. Whereas Figure 12 B, 
iand ii, clearly shows temporal synchronization along the diago- 
nal (and off-diagonals), the same result is not visually discernible 
in Figure 12 Biii. 

We found highly significant differences between the Vim/ 
Vop and STN in terms of the proportions of synchronous 
pairs. A significantly higher proportion of Vim/Vop pairs were 
synchronous than STN pairs (two-tailed Fisher’s exact test, 
p <0.001). 

It has been reported that the level of tremor in parkinsonian 
patients is positively correlated to the degree of pairwise syn- 
chrony among STN cells (Levy et al., 2000). To test the relation- 
ship between tremor tuning and local synchrony, we compared 
the subpopulation of both Vim/Vop and STN neurons that were 
synchronous with at least one other neuron in their respective 
ensembles to the subpopulation of neurons tuned to hand tremor 
(PEPH method). Only neurons fulfilling the criteria of both in- 
dividual analyses were considered. The results are shown in Table 
7. The observed proportions are significantly different (two- 
tailed Fisher’s exact test, p < 0.01), indicating a clear interaction 
between tremor tuning and local synchrony. Only units synchro- 
nized to at least one other unit were tuned to tremor, whereas no 
unsynchronized units were tuned to tremor. 
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Figure 12. 


Table 7. Comparison of synchrony (cross-correlation method) and tremor tuning 
(PEPH method) 


Tremor tuning (PEPH method) 


Synchronized to =1 unit Not synchronized 


Tremor tuned 28 0 
Not tremor tuned 137 30 
Discussion 


In this study, we analyzed ensemble activity of human subcortical 
neurons (either Vim/Vop or STN) in 25 patients, recorded in 
patients who performed visually guided hand movements. To 
our knowledge, this constitutes the largest sample of human sub- 
cortical ensemble recordings to date. We quantitatively evaluated 
the representation of motor parameters in these neurons, as well 
as activity thought to be related to pathological states—tremor 
sensitivity, oscillations, and pairwise synchrony. The present 
work also supports our previous proposition that a sufficiently large 
ensemble of subcortical neurons could enable a motor BMI (Patil et 
al., 2004). We further propose that chronic subcortical microelec- 
trode technology could serve as the basis of a new generation of 
neuroprosthetic devices aimed at both monitoring and actively re- 
ducing oscillatory firing and elevated levels of network synchrony. 


A, Example plots of cross-correlation coefficient for three highly synchronous neuron pairs. Bootstrapped simula- 
tions of coefficient shown in red. B, Example normalized JPSTHs for the same three neuron pairs. i, Pair of Vim/Vop cells, Patient. 
ii, Pair of STN cells, Patient H. iii, Pair of Vim/Vop cells, Patient Q. For B, reported time is relative to target appearance. 
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Subcortical encoding of behavior 
Neurons in both subcortical areas (Vim/ 
Vop and STN) were found to encode fea- 
tures of patient motor behavior, both 
voluntary (target tracking) and involun- 
tary (tremor). During voluntary behavior, 
a substantial number of neurons were 
found to be tuned to target appearance, 
movement onset, and movement direc- 
tion (Table 2). This finding likely indi- 
cates that neurons in both structures are 
broadly tuned across multiple modalities 
and muscle groups. This observation is 
consistent with previous studies in which 
STN neurons have been reported to have 
large receptive fields that respond to mul- 
tiple joints (Abosch et al., 2002), with 40% 
of STN cells tuned to movement in a sim- 
ple two-dimensional joystick task (Wil- 
liams et al., 2005). It has been reported 
that 42% of STN neurons respond to pas- 
sive movement of either arm or leg, and of 
these, 25% responded to multiple joint 
movements (Theodosopoulos et al., 
2003), whereas 51% of Vim/Vop neurons 
are tuned to sensory stimuli, with an over- 
lapping 10% of these cells tuned to voli- 
tional movements (Lenz et al., 1990). 
Some thalamic and STN neurons exhib- 
ited bimodal encoding of target and move- 
ment (Fig. 4), demonstrating what appears 
to be superposition of two independent 
neural representations. Furthermore, we 
observed a significant overlap between tun- 
ing to target appearance and to movement 


1 S onset in both structures. Additionally, di- 
| 


rectional tuning strength was positively cor- 
related to movement tuning strength, 
indicating that individual neurons encoded 

oth parameters by exhibiting both 
common-mode and differential activity rel- 
ative to movement onset (Fig. 3). Thus, neu- 
rons from both subcortical areas exhibited rate modulations based 
ona broad superposition of task parameters (Table 1). 

Our analysis of involuntary motor activity (tremor) yielded 
phase histograms demonstrating clear phase-locking between neu- 
ronal activity and tremor periods (Fig. 6). Overall, we report signif- 
icant populations of tremor tuned cells (12.4% for Vim/Vop, 15.9% 
for STN). The literature reports a large range in the prevalence of 
tremor-related cells. For STN, researchers have reported 11% 
(Magarifios-Ascone et al., 2000), 19% (Rodriguez-Oroz et al., 2001), 
and 52% (Amtage et al., 2008). For Vim/Vop, researchers have re- 
ported 34% (Lenz et al., 1988), 35.6% (Zirh et al., 1998), and 51% 
(Hua and Lenz, 2005). These disparities are probably due to differ- 
ences in recording parameters and classification methodology. For 
example, whereas many researchers have classified tremor tuning 
using linear coherence between spike train and recorded EMGs, we 
used kinematic hand position recorded by a haptic glove. 

We found substantial overlap between tremor-tuned neurons 
and those tuned to parameters of voluntary motor behavior (Ta- 
ble 1). Moreover, we also observed a positive correlation between 
tremor tuning strength and directional tuning strength (Vim/ 
Vop cells only). These results are consistent with earlier studies 
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(Magarifios-Ascone et al., 2000; Rodriguez-Oroz et al., 2001) that 
reported a large proportion of tremor-related STN cells to be 
simultaneously related to voluntary movements either through 
motor or sensory loops. 


Elusive relationship between oscillatory activity and tremor 
Within a tremor-relevant frequency range (2.5—7.5 Hz), we ob- 
served a substantial population of Vim/Vop and STN cells exhib- 
iting strong oscillations (Table 4). Furthermore, we found that 
the mean autopower spectra for recorded Vim/Vop cells had a 
higher peak frequency than that of recorded STN cells (Fig. 9B). 
This finding is consistent with reports of higher mean tremor 
frequency in ET patients (4—12 Hz) than in PD patients (3-6 Hz) 
(Deuschl et al., 1998). 

Whereas Rodriguez-Oroz et al. (2001) reported that oscilla- 
tory cells in STN did not represent movements, we found mod- 
erate overlap (Table 1) between Vim/Vop and STN neurons 
exhibiting oscillations and those tuned to voluntary behavioral 
parameters (target, movement, direction). This important find- 
ing indicates that there is not a strict dichotomy between patho- 
logical neurons and those encoding motor signals. 

Similarly, we found overlap but no significant correlation be- 
tween the presence of oscillatory patterns and tremor tuning in 
both subcortical areas. This finding corroborates the claims of 
earlier studies of ventral thalamus and STN (Magnin et al., 2000; 
Rodriguez-Oroz et al., 2001), in which the sets of tremor-related 
neurons and oscillatory neurons show modest intersection. In 
other words, not all tremor-related neurons exhibited oscilla- 
tions, and some oscillatory neurons exhibited no clear associa- 
tion with tremor. 

Furthermore, spectral peak detection methods revealed no 
relationship between spike train and hand acceleration (Fig. 10), 
though there may be an elusive nonlinear or nonstationary relation- 
ship between neuronal oscillations and hand tremor. We hypothe- 
sized that for some cells, spike train and tremor comodulated with 
relatively large bandwidth. Our heterodyne decoding results suggest 
that a substantial number of cells in both subcortical areas may be 
tremor-tuned in this manner (Table 5). Overall, no consensus in the 
field has been reached regarding the definitive relationship between 
oscillatory behavior and tremor tuning, and our results support the 
idea that pathological oscillations are idiopathic across both neurons 
and patients. 


Network synchrony 

Dopamine depletion in PD has been reported to promote neuro- 
nal synchrony within the basal ganglia (Heimer et al., 2006). In 
this study, a high percentage of neuronal pairs from both subcor- 
tical areas exhibited synchronous behavior (Table 6). This is gen- 
erally consistent with reports of pairwise synchrony in the 
majority of analyzed STN pairs (Levy et al., 2000, 2002). How- 
ever, we observed high levels of functional synchrony in neuronal 
pairs much further apart than the submillimeter separation in 
Levy et al. (2000, 2002). Furthermore, Vim/Vop neuron pairs 
(from ET patients) exhibited significantly higher synchrony than 
STN pairs. 

Remarkably, only cells exhibiting synchrony with another cell 
in the ensemble were found to be tremor-tuned; no unsynchro- 
nized cells were tremor-tuned (Table 7). This finding corrobo- 
rates reports that the prevalence of STN pairs synchronized at 
high frequencies is correlated to the degree of parkinsonian 
tremor (Levy et al., 2000) and that patients without observable 
tremor do not exhibit high-frequency STN synchrony (Levy et 
al., 2002). Others have reported correlations between STN syn- 
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chrony and bradykinesia/rigidity (Weinberger et al., 2009). Our 
findings have extended Levy et al.’s (2000, 2002) results to com- 
binatorial pairs in neuronal ensembles from both STN and Vim/ 
Vop. It remains unclear to what degree network synchrony is 
indicative of tremor pathology, although the effective disruption 
of unstable network activity by DBS stimulation suggests a 
connection. 


Chronic subcortical ensemble recordings 

Acute intraoperative recordings exhibit instability and other lim- 
itations, whereas long-term recordings from subcortical struc- 
tures will offer greater signal quality following the recovery of the 
electrode-tissue interface. Thousands of subcortical DBS implan- 
tations are performed every year with minimal risk (Bronstein et 
al., 2011). Furthermore, long electrode tracts have been shown to 
yield more stable extracellular recordings (Porada et al., 2000; 
Kriiger et al., 2010) by mitigating the problem of electrode mi- 
cromotion seen in cortical implants. 

We suggest that chronic subcortical ensemble recordings may 
bring about the viability of subcortical BMI systems, first dis- 
cussed in our previous study (Patil et al., 2004). Our reported 
offline prediction results from our best Vim/Vop and STN ses- 
sions are comparable to those reported from selected rhesus ma- 
caque cortical regions with similar neuron count (Wessberg et al., 
2000; Carmena et al., 2003). However, hyperbolic extrapolation 
(Wessberg et al., 2000) suggests an ensemble size greater than 100 
to achieve prediction fidelity above R* = 0.9. This would of 
course necessitate the design of microelectrode arrays for 
cannula-based implantation with more recording sensors and 
demonstrable long-term safety and recording efficacy. 

We propose that clinical studies using chronic ensemble re- 
cordings in humans will permit both the continued study of the 
neurophysiological mechanisms involved in motor control as 
well as long-term monitoring of pathological activity. Chronic 
recordings will facilitate continuous examination of changes in 
tuning, oscillations, and synchrony as a function of the patient’s 
symptomatic state both on and off treatment. Specifically, this 
wealth of electrophysiological data may well be used to instruct 
the improvement of closed-loop DBS systems that are currently 
in initial stages of development (Rosin et al., 2011; Rouse et al., 
2011). 

We have shown that within the STN and Vim/Vop thalamus, 
there is an idiopathic mixture of pathology and behavior tuning; 
these sites are worthy of more targeted treatment than the current 
dominant approach of delivering high-frequency DBS through 
macroelectrodes. Subcortical ensembles remain an untapped re- 
source, with the potential to advance both neuroscience and neu- 
rorehabilitation alike. 
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Brain-machine interfaces (BMIs) provide a new assistive strategy aimed at restoring mobility in 

: severely paralyzed patients. Yet, no study in animals or in human subjects has indicated that long- 

: term BMI training could induce any type of clinical recovery. Eight chronic (3-13 years) spinal cord 

: injury (SCI) paraplegics were subjected to long-term training (12 months) with a multi-stage BMI- 
based gait neurorehabilitation paradigm aimed at restoring locomotion. This paradigm combined 
intense immersive virtual reality training, enriched visual-tactile feedback, and walking with two EEG- 

: controlled robotic actuators, including a custom-designed lower limb exoskeleton capable of delivering 

: tactile feedback to subjects. Following 12 months of training with this paradigm, all eight patients 

: experienced neurological improvements in somatic sensation (pain localization, fine/crude touch, and 
proprioceptive sensing) in multiple dermatomes. Patients also regained voluntary motor control in key 
muscles below the SCI level, as measured by EMGs, resulting in marked improvement in their walking 
index. As a result, 50% of these patients were upgraded to an incomplete paraplegia classification. 
Neurological recovery was paralleled by the reemergence of lower limb motor imagery at cortical level. 
We hypothesize that this unprecedented neurological recovery results from both cortical and spinal 
cord plasticity triggered by long-term BMI usage. 


Spinal Cord Injury (SCI) rehabilitation remains a major clinical challenge, especially in cases involving chronic 

complete injury. Clinical studies using body weight support systems’”, robotic assistance’, and functional elec- 

trostimulation of the leg** have proposed potential solutions for assisting SCI patients in walking”*. Yet, none of 

these approaches have generated any consistent clinical improvement in neurological functions, namely soma- 
: tosensory (tactile, proprioceptive, pain, and temperature) perception and voluntary motor control, below the 
: level of the spinal cord lesion. 
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Since the first experimental demonstrations in rats®, monkeys!*"!, and the subsequent clinical reports in 


humans!*""4, brain-machine interfaces (BMIs) have emerged as potential options to restore mobility in patients 
who are severely paralyzed as a result of spinal cord injuries (SCIs) or neurodegenerative disorders'*. However, 
to our knowledge, no study has suggested that long-term training associating BMI-based paradigms and phys- 
ical training could trigger neurological recovery, particularly in patients clinically diagnosed as having a com- 
plete SCI. Yet, in 60-80% of these “complete” SCI patients, neurophysiological assessments'®!” and post-mortem 
anatomical'® studies have indicated the existence of a number of viable axons crossing the level of the SCI. This 
led some authors to refer to these patients as having a “discomplete” SCI” and predict that these remaining axons 
could mediate some degree of neurological recovery. 

For the past few years, our multidisciplinary team has been engaged in a project to implement a multi-stage 
neurorehabilitation protocol - the Walk Again Neurorehabilitation (WA-NR) - in chronic SCI patients. This 
protocol included the intensive employment of immersive virtual-reality environments, combining train- 
ing on non-invasive brain-control of virtual avatar bodies with rich visual and tactile feedback, and the use of 
closed-loop BMI platforms in conjunction with lower limb robotic actuators, such as a commercially available 
robotic walker (Lokomat, Hocoma AG, Volketswil, Switzerland), and a brain-controlled robotic exoskeleton, 
custom-designed specifically for the execution of this project. 

Originally, our central goal was to explore how much such a long-term BMI-based protocol could help SCI 
patients regain their ability to walk autonomously using our brain-controlled exoskeleton. Among other innova- 
tions, this device provides tactile feedback to subjects through the combination of multiple force-sensors, applied 
to key locations of the exoskeleton, such as the plantar surface of the feet, and a multi-channel haptic display, 
applied to the patient’s forearm skin surface. 

Unexpectedly, at the end of the first 12 months of training with the WA-NR protocol, a comprehensive neu- 
rological examination revealed that all of our eight patients had experienced a significant clinical improvement 
in their ability to perceive somatic sensations and exert voluntary motor control in dermatomes located below 
the original SCI. EEG analysis revealed clear signs of cortical functional plasticity, at the level of the primary 
somatosensory and motor cortical areas, during the same period. These findings suggest, for the first time, that 
long-term exposure to BMI-based protocols enriched with tactile feedback and combined with robotic gait train- 
ing may induce cortical and subcortical plasticity capable of triggering partial neurological recovery even in 
patients originally diagnosed with a chronic complete spinal cord injury. 


Methods 

Eight paraplegic patients, suffering from chronic (>1 year) spinal cord injury (SCI, seven complete and one 
incomplete, see Fig. 1A, Supplementary Methods Inclusion/exclusion Criteria), were followed by a multidisci- 
plinary rehabilitation team, comprised of clinical staff, engineers, neuroscientists, and roboticists, during the 
12 months of 2014. Our clinical protocol, which we named the Walk Again Neurorehabilitation (WA-NR), was 
approved by both a local ethics committee (Associacdo de Assisténcia a Crianga Deficiente, Sao Paulo, Sao Paulo, 
Brazil #364.027) and the Brazilian federal government ethics committee (CONEP, CAAE: 13165913.1.0000.0085). 
All research activities were carried out in accordance with the guidelines and regulations of the Associagao de 
Assisténcia a Crianga Deficiente and CONEP. Each participant signed written informed consent before enrolling 
in the study. The central goal of this study was to investigate the clinical impact of the WA-NR, which consisted 
of the integration between traditional physical rehabilitation and the use of multiple brain-machine interface 
paradigms (BMI). This protocol included six components: (1) an immersive virtual reality environment in which 
a seated patient employed his/her brain activity, recorded via a 16-channel EEG, to control the movements of a 
human body avatar, while receiving visuo-tactile feedback; (2) identical interaction with the same virtual envi- 
ronment and BMI protocol while patients were upright, supported by a stand-in-table device; (3) training on a 
robotic body weight support (BWS) gait system on a treadmill (Lokomat, Hocoma AG, Switzerland); (4) training 
with a BWS gait system fixed on an overground track (ZeroG, Aretech LLC., Ashburn, VA); (5) training with a 
brain-controlled robotic BWS gait system on a treadmill; and (6) gait training with a brain-controlled, sensorized 
12 degrees of freedom robotic exoskeleton (see Supplementary Material). 

In all cases except components 3 and 4 above, patients received continuous streams of tactile feedback from 
either the virtual (body avatar) or robotic devices (Lokomat and exoskeleton) via a haptic display (consisting of 
arrays of coined shaped vibrators) applied to the skin surface of the patient’s forearms. Tactile stimulation on the 
forearm was given in accordance with the rolling of the ipsilateral virtual or robotic feet on the ground. 

Two BMI strategies were employed throughout training. Initially, patients were required to imagine move- 
ment of the arms to modulate EEG activity so that they could generate high level motor commands such as ‘walk’ 
or ‘stop. Once patients mastered this first method, they learned to use EEG signals to control individual avatar/ 
robotic leg stepping by imagining movements of their own legs (see Supplementary Material for details). For the 
first paradigm, after the selection of the correct state, patients confirmed their choice by performing an isometric 
contraction of the triceps muscle. 

Three lower limb actuators were used for the BMI experiments: a simulated 3D virtual avatar, the Lokomat 
gait trainer (Hocoma AG, Switzerland); and a custom built exoskeleton. The virtual avatar was simulated in 
MotionBuilder (Autodesk MotionBuilder 2014) and visualized from first person perspective using an immersive 
head mounted displayer (Oculus Rift, Oculus VR). The Lokomat is a robotic gait device that uses a body weight 
support (BWS) system integrated with a treadmill. Finally, a custom built exoskeleton was used with our patients. 
The exoskeleton had autonomous power, self-stabilization, and full lower limb hydraulic actuation. It was built 
to accommodate a wide weight range of SCI patients (50-80 kg) while not necessitating the use of crutches (See 
Supplementary Movie $11). Our exoskeleton was used in conjunction with a ZeroG!? system containing an over- 
ground BWS system that rides along an overhead fixed track. In this setup, there were no mechanical barri- 
ers between the patient and physical therapist. For this reason, the walking setup offers more challenges to the 
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Figure 1. Methodology. (A) Cumulated number of hours and sessions for all patients over 12 months. We report 
cumulated hours for the following activities: classic physiotherapy activities (e.g. strengthening/stretching), 
gait-BMI-based neurorehabilitation, one-to-one consultations with a psychologist, periodic measurements 

for research purposes and routine medical monitoring (vital signs, etc.). (B) Neurorehabilitation training 
paradigm and corresponding cumulated number of hours for all patients: 1) Brain controlled 3D avatar with 
tactile feedback when patient is seated on a wheelchair or 2) in an orthostatic position on a stand-in-table, 3) 
Gait training using a robotic body weight support (BWS) system on a treadmill (LokomatPro, Hocoma), 4) 
Gait training using an overground BWS system (ZeroG, Aretech). 5-6) Brain controlled robotic gait training 
integrated with the sensory support of the tactile feedback at gait devices (BWS system on a treadmill or the 
exoskeleton). (C) Material used for the clinical sensory assessment of dermatomes in the trunk and lower 
limbs: to evaluate pain sensitivity, examiner used a pin-prick in random positions of the body segments. Nylon 
monofilaments applying forces ranging between 300 to 0.2 grams on the skin, were used to evaluate patients’ 
sensitivity for crude to fine touch. Dry cotton and alcohol swabs were used to assess respectively warm and cold 
sensation. Vibration test was done using a diapason on patients’ legs bone surface. Deep pressure was assessed 
with an adapted plicometer in every dermatome. 


subjects, in comparison to off-the-shelf devices, by requiring patients to be in charge of postural and trunk con- 
trol, upper limb strength and dynamic balance. Further gait training was performed by having subjects utilize a 
lower limb orthosis and walking assistive devices (hip-knee-ankle-foot orthosis or ankle-foot orthosis with knee 
extension splint and wheeled triangular walker). 

Throughout the application of our protocol, the complexity of activities was increased over time to ensure 
cardiovascular system stability and better patient postural control; starting with orthostatic training at a 
stand-in-table and progressing all the way to the different gait training robotic systems””~. In addition to rou- 
tine general clinical evaluations (i.e. cardiovascular function, intestinal and urinary emptying, skin inspection, 
spasticity handling), before and after every activity, and a long-term treatment of osteoporosis, multiple clini- 
cal evaluations were periodically performed in order to identify possible changes in the neurological status of 
the SCI and to assess psychological and physical conditions. Such clinical evaluation started on the first day 
patients began training (Day 0), and were repeated after 4, 7, 10, and 12 months. Clinical evaluations included: 
the American Spinal Injury Association (ASIA) Impairment Scale (International Standards for the Neurological 
Classification of Spinal Cord Injury’), the Semmes-Weinstein Monofilament Test”* (Fig. 1B), the evaluation of 
temperature, vibration, proprioception and deep pressure sensitivity, a muscle strength test (Lokomat L-force 
Evaluation)”*”*, the Thoracic-Lumbar Scale for trunk control assessment”’, Walking Index Spinal Cord Injury 
II (WISCI)”8, Spinal Cord Independence Measurement III (SCIM)”, McGill Pain Questionnaire™ and Visual 
Analogue Scale (VAS)*!” for pain evaluation, the range of motion of lower limb joints of the Medical Research 
Council scale**, Modified Ashworth Scale* and the Lokomat L-stiff Evaluation for spasticity**, the World Health 
Organization Quality of Life Assessment Instrument-Bref (WHOQoL-Bref)*%, the Rosenberg Self-Esteem Scale*’, 
and the Beck Depression Inventory (BDI)**. 
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Lesion GS | Time since the 
Gen | Age | grade | Right | Left | lesion(years) | Etiology 

Patient 1 F 32 ASIA A Tll T10 13 Closed trauma 
Patient 2 M 26 ASIA B T4 T4 6 Closed trauma 
Patient 3 M 32 ASIA A T10 Tll 5 Open injury 

Patient 4 M 38 ASIA A T8 T8 5 Closed trauma 
Patient 5 M 36 ASIA A 17 17 3 Closed trauma 
Patient 6 M 29 ASIA A T4 T4 8 Closed trauma 
Patient 7 M 27 ASIA A 17 TS 6 Closed trauma 
Patient 8 F 29 ASIA A Tll Tll ll Closed trauma 


Table 1. Patients’ demography. 


Throughout training, the potential occurrence of functional cortical plasticity was evaluated through lon- 
gitudinal analyses of EEG recordings. For this, patients were instructed to imagine movements of their own 
legs while EEG signals from 11 scalp electrodes were recorded over the leg primary somatosensory and motor 
cortical areas. Independent Component Analysis (ICA,*’) was employed to determine potential cortical sources, 
represented by individual independent components (ICs), of novel leg representations in the primary motor and 
somatosensory cortices and to detect functional changes of these representations over time. To evaluate brain 
dynamics modulation, before and after many months of training, we calculated for each IC the Event Related 
Spectral Perturbations (ERSPs) with respect to a baseline of 1 second prior to the event and normalized by the 
average power across trials at each frequency. Event Related Potentials (ERPs), sampled from two EEG electrodes 
located over the leg representation area, averaged over all patients, before and after training, were also calculated 
and used for statistical comparison. 


Results 

Altogether, the eight paraplegic patients enrolled in this protocol (Table 1 for patient’s demography) participated 
in a total of 2,052 sessions, for a collective total of 1,958 hours, divided into multiple phases of neurorehabilitation 
training (Fig. 1A). Figure 1B details the distribution of hours of the six stages employed in our WA-NR protocol. 

As part of the neurological evaluation, we periodically tested all patients’ sensitivity to fine touch, pain, tem- 
perature, vibration, pressure, and proprioception (Fig. 1C). Figure 2A describes the individual improvement of 
each patient, in number of dermatomes, for tactile (Semmes- Weinstein monofilament test) and pain sensitivity 
(ASIA sensory evaluation) for all eight patients at the end of 10 month of training. The left graph shows the 
improvement of normal sensation, while the right graph depicts the improvement in altered sensation (hyper or 
hypoesthesia). In each of the graphs, improvement is described in terms of number of dermatomes, i.e. the body 
area that receives sensory innervation from a given spinal nerve root. In this analysis, we measured the extent of 
the Zone of Partial Preservation (ZPP), i.e. the dermatomes and myotomes (the set of muscles innervated by one 
spinal nerve) caudal to the neurological level of injury that remain partially innervated’*. The ZPP applies only 
for complete SCI patients. 

Figure 2B displays the patients’ average improvement, in number of dermatomes below the SCI, for both 
tactile and pain sensation after 10 months of training. Inspection of these two figures reveals significant improve- 
ment in every monofilament and pin tested for all patients. Although we observed significant improvements in 
both the zone of normal sensation and in the ZPP, the largest gains (5.1 +/— 0.9 dermatomes) were consistently 
observed in the latter in every patient. In terms of tactile sensitivity, the largest improvements in number of 
dermatomes (1 normal/2 altered both pink and orange) were observed for the pink (300 g) and orange monofil- 
aments (10g). For the orange monofilament, the improvement was 1 dermatome for normal sensory area and 2 
dermatomes for the ZPP area. Figure 2C depicts two of the patients who exhibited the best improvement in both 
fine tactile and nociceptive perception. Notice that the improvement was larger for the latter and includes many 
dermatomes below the SCI level. 

Throughout the study, all patients reported some type of pain sensation below the SCI level, as measured by 
the McGill’s questionnaire. Yet, patients experienced some difficulty in reporting the exact location of such pain. 
This finding corroborates the results described by Demirel et al.*° who showed that 68% of their SCI patients 
reported pain in the lower limbs. As the training progressed, the patients’ self-reported pain intensity in daily life - 
as measured by the visual analogue scale (VAS,**) - decreased on average from 2.0 to 1.3 (maximum 10, n=8) 
while perception of pain at the moment of the evaluation (Present Pain Intensity evaluation integrates McGill 
Questionnaire*°) stayed very low (sum over all patients = 3, with a maximum possible score = 40) throughout 
the year. An improvement in patients’ ability to perceive and relate more effectively the occurrence of lower limb 
pain as well the pain location in the body was observed at the end of the study. Supplementary Movie S1 shows 
the mean improvement for pain sensation over the 1 year training period. 

Figure 2D-I describes the mean evolution of every somatosensory parameter (pain, tactile, temperature, pres- 
sure, vibration, and proprioception) measured over the time course of our study. These graphs revealed that we 
did not observe any significant improvement in temperature sensitivity (Fig. 2F). Conversely, pressure sensitivity 
improved by 1 (normal) and 2 (altered) dermatomes, between the 4" and 10" month (Fig. 2G). Figure 2H indi- 
cates that the average sensitivity to vibration also improved, mainly for the hip joint (anterior superior iliac spine), 
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Figure 2. Sensory improvement after neurorehabilitation training. (A) Left table reports the improvement 
for altered sensation (hyper or hypoesthesia) and the right table reports normal sensation. Number of 
dermatomes recovered after 10 months training compared to baseline (recorded at day 1 of training) for pain 
and tactile sensory modalities for all patients and body sides. Sensory modalities are reported by pain, crude 
touch (applied with a 300 gr. monofilament), and gradually more selective touch (applied with 10 gr. to 0.2 gr. 
monofilaments). The brightness of each colored square represents the magnitude of improvement, considering 
each monofilament and pin employed (lightest represents highest improvement). (B) Average sensory 
improvement (mean +/— SEM over all patients) after 10 months training. (C) Example of improvement in 

the Zone of Partial Preservation for sensory evaluation for two patients. (D-G) Mean +/— SEM of lowest 
dermatome with normal (red) or altered (blue) sensation for (D) Pain (E) Tactile - crude touch, pink 
monofilament (F) Temperature and (G) Pressure on the body calculated over all patients. From (D-G), y-axis 
exhibits dermatomes in a cranio-caudal order, following the anatomic sequence. Baseline was recorded during 
the year following the injury, time 0 represents the starting day of our training. For each modality we show the 
average over raw (top graph) and z-scored data (lower graph). P-values for Wilcoxon rank sum test are reported 
on z-scored data (*p < 0.05, **p <0.01, ***p < 0.001). (H) Mean score for perception of vibration on eight leg 
bones presented (most proximal to most distal order). Score convention was the following: 0 for no sensation, 1 
for altered sensation and 2 for normal sensation. (I) Mean score for proprioception (0: absent, 1: present) over 
lower limb joints. Note that measurements for temperature, vibration and proprioception were introduced 4 
months after the beginning of the training. 


but also for the knee and the ankle. In all patients, improvement went from almost total absence to mid-level 
vibration sensation (see Figure S1 for details per patient). 

‘The average proprioception sensation across all patients also improved significantly during the period between 
the 4""-12" months (Fig. 21). This improvement was mainly observed at the hip level, since patients became 
capable of distinguishing hip flexion and extension for both the right and left sides. Secondary improvement was 
detected at the knee and ankle, from the 7" month to the 12" month (see Figure S2 for details per patient). 

Improvement in motor function was also extensively documented in our eight patients. Figure 3A depicts 
the evolution of the ASIA motor protocol assessment for all patients, considering five key lower limb muscles: 
hip flexors (rectus femoris proximal portion), knee extensors (rectus femoris distal portion), ankle dorsiflexors 
(tibialis anterior), ankle plantar flexors (gastrocnemius), and long toe extensors. This analysis revealed that every 
patient exhibited some degree of improvement in voluntary muscle contraction below the SCI level. Indeed, seven 
patients experienced an improvement, ranging from two to multiple key muscles below the SCI level. The bottom 
shelf of Figure 3A depicts the average improvement for all eight patients for 11 muscles (five key muscles and six 
secondary muscles: gluteus medius, gluteus maximus, hip adductor, medial and lateral hamstrings, and long toe 
flexor) ranged in a proximal to distal order. Overall, the strongest motor improvement was observed in the six 
most proximal muscles. 

Motor recovery was also documented through multi-channel surface EMG recordings (Fig. 3B,C). For this 
evaluation, patients were instructed verbally by a physiotherapist to alternate movements of their left leg with 
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Figure 3. Clinical and neurophysiological assessments of lower limb motor recovery. (A) Detail of key 
muscle improvement per patient, according to clinical evaluation (ASIA) for all patients and the average over 
all 8 patients for key and secondary lower limb muscles listed in a proximal to distal order (secondary muscles 
are in italic). Patients that changed classification to ASIA C after 12 months of training have 2 lines rings around 
their names. (B) Details of the EMG recording procedure in SCI patients. (B1) Raw EMG for the right gluteus 
maximus muscle for patient P1 is shown at the top of the topmost graph. The lower part of this graph depicts 
the envelope of the raw EMG, after the signal was rectified and low pass filtered at 3 HZ. Gray shaded areas 
represent periods where the patient was instructed to move the right leg, while the blue shaded areas indicate 
periods of left leg movement. Red areas indicate periods where patients were instructed to relax both legs. (B2) 
All trials over one session were averaged (mean +/— standard deviation envelopes are shown) and plotted 

as a function of instruction type (gray envelope = contract right leg; blue = contract left leg; red = relax both 
legs). (B3) Below the averaged EMG record, light green bars indicate instances in which the voluntary muscle 
contraction (right leg) was significantly different (t-test, p < 0.01) than the baseline (periods where she/he 

was instructed to relax both legs). Dark green bars depict periods in which there was a significant difference 

(p <0.01) between muscle contraction in the right versus the left leg. (C) EMG envelops and t-tests for all 
recording sessions, involving 4 muscles, for all 8 patients: left and right gluteus maximus (GMx) and reto 
femoral proximal (RFP) muscles. Color convention and figure organization follows the one of panel B. Data was 
collected after 7 months of training for all patients and for all but patients P2 and P8 after 12 months. 


movements of their right leg, and periods in which neither leg should be moved (Fig. 3B). The first EMG ses- 
sion was recorded after 7 months of training (all patients), then a second session was obtained at 12 months (all 
patients except P2 and P8). Since all patients were completely paralyzed below the level of injury, none of them 
exhibited any motor activity below the level of their SCI at the onset of the training. However, 7 months into train- 
ing, EMG recordings revealed that all patients started to show signs of motor recovery, indicated by their ability 
to voluntarily control at least one muscle below the level of the SCI (Fig. 3C). By the 12 month of training, this 
motor recovery had stabilized and, in most cases, improved significantly. Figure 3B,C shows that patient P1 exhib- 
ited the strongest and most consistent voluntary contractions (Fig. 3B,C, Supplementary Movies S3, S4, S8, $9) in 
both left and right gluteus maximus (GMx) and reto femoral proximal (RFP) muscles (contraction significance, 
t-test, p< 0.01, is indicated in light green under each graph). This patient’s motor control was clearly selective, 
i.e., a stronger contraction of the right GMx and RFP was observed when the patient was instructed to contract 
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| RightLeg7 | RightLeg | LeftLeg7 | Left Leg 12 | 
monthsof | 12months | monthsof | months of 

Patient | training | oftraining | training | training 
Pl 17.7 21.14 16.85 16.12 
P2 111 1.17 0.95 1.31 
P3 2.76 4.42 4.34 4.6 
P4 0.47 1.48 0.9 1.07 
P5 0.93 43 1.22 2.13 
P6 0.25 0.6 0.3 0.8 
P7 0.45 0.89 0.58 12 
P8 5.01 9.36 


Table 2. Torque (N-m) measurement 7 and 12 months after beginning of training. Recordings were 
performed while patient was standing in the Lokomat (L-Force measurement). Patient was instructed by the 
physiotherapist to contract left or right leg (hip flexion). Peak of torque over six repetitions is reported. 


the right leg, while left GMx and RFP contractions were produced following the command to contract the left 
leg (see selectivity significance reported in dark green under each figure, t-test, p< 0.01). Patient P3 exhibited 
strong significant contractions in both left and right GMx muscles (Fig. 3C, Supplementary Movie S4). However, 
reliable RFP contractions, absent at the 7 month, appeared and became consistent at the end of the 12" month of 
training. The same evolution trend in motor control was observed in three other patients, P4, P5, and P7: while at 
the 7" month these patients exhibited weak or no voluntary GMx muscle contractions, after 12 month of training 
such contractions became evident in the EMG recordings (bilateral for P4 and P5, left side in P7, Fig. 3C). Finally, 
patients P2 and P8 also displayed significant voluntary muscle contractions in their GMx muscles during the 7" 
month of training. P8 also exhibited selective contractions in both her RPF muscles. Altogether, these longitudi- 
nal EMG recordings suggest that a sustained and long-term training protocol may be required to trigger motor 
improvement in ASIA A patients. 

Motor contractions were also documented through the employment of the Lokomat torque sensors during the 
same type of task used for EMG measurements (Table 2). Patient P1 showed the best recovery at 21.14N-m (12 
month, right leg) for hip flexion (thus RFP contraction), followed by patient P8 at 9.36 N-m (7 month), and P3 
and P5 at 4.6 and 4.3 N-m. Patients P2, P4, P6, P7 produced torques in the 0.8-1.5 N-m range. 

Figure 4A and Supplementary Movie S2 illustrate the temporal progression of the average motor recovery 
observed in our patients, from month 0 to month 12, considering both key ASIA muscles and secondary lower 
limb muscles. Notice that this motor recovery clearly progressed from proximal to distal muscles, being more 
pronounced at the level of the hip joint. Thus, the muscles that exhibited the best recovery were: the gluteus max- 
imum (maximum score 3; average score over all patients 1.56 and 1.5 for right and left side respectively), and glu- 
teus medius (maximum score 2; average score 1.25 and 1.06), the proximal portion of the rectus femoris (max 3; 
average 1.56 and 1.5), and the hip adductors (max 2; average 1.18 and 1.06). At the level of the knee joint, the 
greatest improvements were observed in the medial and lateral hamstring (max 1; average 0.38 and 0.31), and the 
distal portion of the rectus femoris (max 2; average 1.06 and 0.88). At the ankle level, the greatest motor improve- 
ment was located in the sural triceps (max 1.5; average 0.31 and 0.25) and the anterior tibialis (max score 1; 
average 0.5 and 0.38). 

Overall, the global patterns of both sensory and motor recovery indicated a proximal to distal progression, 
below the level of the patients’ SCI that evolved to include sacral roots. Supplementary Movies S3-S10 shows 
patients’ clear lower limb contractions while in a hanging or lying positions after 1 year of training with the 
WA-NR protocol. As a result of this sensory and motor recovery, Fig. 4B depicts the progression over time of the 
ASIA classification showing that 50% of our patients (n= 4) changed their ASIA classification in 12 months of 
training: three of these patients moved from ASIA A to ASIA C and one patient moved from ASIA B to ASIA C. 

To further document this recovery, Fig. 4C displays the individual patient improvement in thoracic-lumbar 
strength and stability measured in different positions — seated and laying down - and static and dynamic balance. 
Between the 7" and 10% months, five of our patients improved significantly in this type of motor control. To illus- 
trate how this motor recovery functionally impacted the patients, Fig. 4D depicts the progression of their Walking 
Index for SCI over the last 5 months of training. As one can see, all patients showed significant improvement in 
assisted walking skills; two patients increased their performance by six levels, four patients by five levels, and 
two more patients by three levels. For example, while Patient 1 was initially not even able to stand using braces 
when placed in an orthostatic posture (score 0), after 10 months of training the same patient became capable 
of walking using a walker, braces and the assistance of one therapist (score 6). At this stage, this patient became 
capable of producing voluntary leg movements mimicking walking, while suspended overground in the Lokomat 
(Supplementary Movie S8). The same patient produced close to 20N*m of hip flexion force while in the Lokomat 
(Table 2). In another example, Patient 7, started at score 6 and progressed all the way to score 12, which means 
that he/she was capable of walking with two crutches and lower limb orthoses (hip-knee-ankle-foot orthoses), 
while requiring no assistance by a therapist. 

In addition to the partial recovery in neurological functions, our patients also exhibited improvements in 
gastrointestinal function and their overall skin condition. Figure 4E plots the monthly evolution of both the mean 
number of standing/walking hours and the z-scored mean frequency of bowel functioning. The top right graph 
in Fig. 4E shows that these latter two variables are highly correlated (r = 0.72, n= 10 measurements). Notice 
that peak bowel function was reached 3 months after the training starting. During the patient's vacation period 
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A. Motor improvement over time (ASIA scale) 
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(12) Ambulates with two crutches, with braces and 
no physical assistance, 10 meters. 

(10) Ambulates with one cane/crutch, with braces 
and physical assistance of one person, 10 meters. 
(8) Ambulates with walker, no braces and 
physical assistance of one person, 10 meters. 

(6) Ambulates with walker, with braces and 
physical assistance of one person, 10 meters. 

(4) Ambulates with parallel bars, no braces and 
physical assistance of one person, 10 meters. 

(2) Ambulates with parallel bars, with braces and 
physical assistance of two people, 10 meters. 

(0) Patient is unable to stand and/or 

participate in assisted walking. 


Figure 4. Clinical and functional improvements. (A) Graphical representation of clinical assessment of 
motor strength, calculated with the ASIA protocol, for key muscles and other muscles in the lower limb (mean 
over all patients). The scale for a given muscle goes from complete transparency for no muscle activation 
(muscle strength score 0) to contraction against gravity (score 3). Time 0 of x-axis means day 1 of the neuro- 
rehabilitation training. (B) Patients with ASIA classification improvements: four patients changed ASIA 
classification over the course of the neurorehabilitation training, three moved from ASIA A to C and one 
moved from ASIA B to C. ASIA A is characterized by absence of both motor and sensory functions in the 
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lowest sacral area, ASIA B by the presence of sensory functions below the neurological level of injury, including 
sacral segments $4-S5 and no motor function is preserved more than three levels below the motor level on 
either side of the body, ASIA C by the presence of voluntary anal sphincter contraction, or sacral sensory 
sparing with sparing of motor function more than three levels below the motor level, majority of key muscles 
have muscle grade less than 3”°. (C) Thoracic-lumbar control scale evaluates quantitatively motor skill of the 
thoracolumbar region. Score ranges between 0 and 65. It has 10 items that considers supine, prone, sitting and 
standing postures. In the present study, the last item (orthostatic position) was scored 0 due to the limitations 
of the pathology. (D) Functional assessment of autonomy in walking given by the Walking Index for Spinal 
Cord Injury scale. The scale ranges between 0, for a patient who is unable to stand and/or to participate in 
assisted walking, to 20 for a patient who ambulates 10 meters with no walking devices, no braces and no 
physical assistance. (E) Correlation between average time spent in a standing position in orthostatic or gait 
training (mean +/— SEM, values are average hours per month) and mean frequency for bowel function (values 
calculated per month and z-scored per patient). 


(months 4-5), lack of standing/walking was correlated with a significant reduction in bowel function. Upon 
restarting of the standing/walking training, bowel function increased again. 

EEG measurements were employed to investigate potential cortical functional reorganization that could 
correlate with the type of sensorimotor improvement observed during training. An Independent Component 
Analysis (ICA*’) was applied to 11 EEG channels both at the beginning, and after 8-10 months of training. 
Briefly, the ICA algorithm isolates maximally independent sources from multi-channel EEG signals to both dis- 
card non-brain signals (such as muscle artifact or noise) and identify the spatial location of distinct brain-derived 
signals contained in the overall EEG recording. 

At the onset of training, in three out of seven patients instructed to imagine moving their own legs, we could 
only identify a total of four significant independent components (sum for all patients) in the putative leg rep- 
resentation area of the primary somatosensory (S1) and motor (M1) cortices. The top shelf of Fig. 5A depicts 
the projection of these four components in the S1/M1 region (in a top and coronal slice) and the corresponding 
event related spectrogram perturbation (ERSP) found for each component is shown in Fig. 5B. The bottom shelf 
of Fig. 5A (respectively 5B for the ERSP) displays the same information, obtained after 8-10 months of training. 
Notice that at this point, a total of 12 significant components (sum for all patients) could be isolated in the leg 
area of the primary sensorimotor cortex of all seven patients who were asked to imagine locomotion movements. 

Further analysis of the ERSP revealed the electrophysiological origins of the four independent components 
identified early in training in three patients (Fig. 5B, top shelf). They corresponded to the presence of desynchro- 
nization of beta waves (16-20 Hz) for Patients 4 and 5, a clear power reduction in mu rhythm (7-12 Hz, first panel 
left) in Patient 5, and a smaller desynchronization of mu in Patient 6. 

In healthy subjects, mu rhythm desynchronization is observed during motor imagery*!”, Desynchronization 
of beta waves (16.5-20 Hz) is also observed during the preparation period, prior to the execution of voluntary 
limb movements‘?*, as well as during motor imagery“®. 

The bottom shelf of Fig. 5B illustrates the changes that occurred in the ERSP after 8-10 months of training. 
At that point, a total of 12 Independent Components (IC) could be isolated in the leg representation area of the 
S1/M1 of all seven patients analyzed. In 4 out of 12 (P1 second IC, P2 first IC, P3 first IC and P7), we detected a 
reduction in mu power. Interestingly, the remaining eight ICs depicted a significant desynchronization in both 
mu and beta waves. Altogether, these findings indicate that, after prolonged BMI-based neurorehabilitation 
training, the leg representation area of $1/M1 cortices of all patients exhibited the type of desynchronization of 
beta wave which has been associated with motor imagery in healthy subjects*®. 

Further evidence for functional plasticity taking place over the training period was obtained through an 
event-related potential analysis, considering two central EEG electrodes, located on the leg representation area of 
the primary S1/M1 cortices, for all seven patients analyzed. Figure 5C shows that, at the onset of training, there 
was no significant desynchronization of the EEG (red line) when patients were asked to imagine walking. Eight to 
ten months later, however, a significant EEG desynchronization (green line) was observed when the average for 
all seven patients was considered. 

According to our inclusion and exclusion criteria, all our patients exhibited a complete range of motion 
(ROM) of the joints and a maximal grade of lower limb spasticity of 2 on the Ashworth scale. As our training 
protocol progressed, we observed that all subjects maintained the original complete range of motion and did not 
develop any muscles contractures. Moreover, their level of lower limb spasticity did not increase their perfor- 
mance during the orthostatic or gait training. By using a Lokomat L-stiff test to quantify the level of spasticity 
of hip and knee muscles for flexors and extensors, we observed that, on average, all patients exhibited a reduced 
spasticity level by the end of 12 months. In addition, half of our patients maintained a stable bone mineral density 
index while the other half demonstrated a slight improvement. There was no correlation between bone mineral 
density and neurological improvement. 

The SCIM questionnaire was applied to assess our patients’ level of functional independence to perform daily 
activities and mobility at home and in a community environment. Although all included patients exhibited a 
good level of independence in daily activities at the onset of training (ranging from 64 to 74 where scores ranged 
between 0 (dependent) to 100 (independent), some patients managed to improve their level of functional inde- 
pendence by the end of the training. For instance, two patients experienced an improvement in the frequency of 
bowel function with no need of using auxiliary devices (laxatives); two became more independent in the bath- 
room environment (perineal hygiene, setting towels/napkins and diapers); and two improved their level of inde- 
pendence in the process of transferring from the wheelchair to the toilet and one from the wheelchair to the car. 


SCIENTIFIC REPORTS | 6:30383 | DOI: 10.1038/srep30383 9 


www.nature.com/scientificreports/ 


A. Independent component B. Event related spectrum 
analysis (ICA) perturbation (ERSP) 
&2 B = 
oo > 
nO 8 
ss 2 o 
esc o ° 
ae) 8 
= o~— ic 
Qe = 
oo 
dB 
4 
' Dipoles 
Dipol 
eno © centroid ) 
wi 


8-10 months after 
Onset of training 
(End) 


Frequency [Hz] 


C. Event related potential (ERP) 


Cz CPz 
1 1 — 0-2 months after 
i 20 ; Onset of training 
u 1 
S i — 8-10 months after 
=O as 
\ og : Onset of training 
\ > ? (End) 
1 
=, 1 
L 
0 4 2 0 4 2 
Time [s] Time [s] 


Figure 5. EEG recording. (A) Functional Cortical dynamics over the leg sensory motor area over time. 
Projection of the position of dipoles found by Independent Component Analysis during leg motor imagery. 
Analysis revealed that the number of dipoles observed in the S1/M1 cortex evolved from four at onset of the 
protocol to 12 at the end of training. Two sets of sessions are shown for patients 1 to 7: one recorded in the first 
2 months of training (Onset) and one recorded between the 7" and 9" month of training (End). (B) For the 
onset and end of training, the Event Related Spectral Perturbation (ERSP) is shown for each of the Independent 
Components (IC) depicted in panel A. At the onset of training, one IC was found for P4 and P5, and two for 
patient P6. At the end of training, two ICs were found for each of the following patients: P1, P2, P3, P4 and P5. 
A single IC was identified for patient P6, and one for patient P7; and none for P8. Decrease in power in Beta 
waves (16-5-20 Hz) is associated with muscle contraction; suppression mu wave (7-5-12-5 Hz) are related to 
motor actions*®. (C) Mean event related potential over all patients for two central electrodes (Cz and CPz) for 
Onset and End of training period. Significant desynchronization or synchronization is marked with an “*’. 


Overall, all patients ranked very high on emotional stability and obtained good scores on the quality of life, 
depression and self-esteem assessment, with minimal fluctuations throughout the study. According to individual 
demand, psychological support was increased, but no use of psychiatric medication was required. 


Discussion 
As far as we can tell, this is the first clinical study to report the occurrence of consistent, reproducible, and sig- 
nificant partial neurological recovery in multiple chronic SCI patients. This partial recovery was manifested 
by improvements in both somatic sensations and voluntary motor control, below the level of the spinal cord 
lesions. This sensorimotor improvement was also paralleled by autonomic improvements, such as bowel function. 
Moreover, this is also the first report demonstrating partial clinical neurological improvements in SCI patients 
subjected to long-term training with a BMI-based gait protocol. Up to now, all previous clinical reports involving 
BMIs focused on decoding and control strategies of artificial prosthetic devices using the subject’s own electrical 
brain activity alone’?-'**”“®. In these studies, patients were able to control the movements of artificial devices 
using their brain activity. Yet, none of these studies described any type of neurological recovery as a consequence 
of BMI training. 

A total of eight chronic SCI patients were trained over the course of 12 months in a multi-stage, progres- 
sive neurorehabilitation protocol - the Walk Again Neurorehabilitation (WA-NR) protocol - that employed a 
non-invasive, EEG-based, closed-loop BMI approach. This protocol required that patients brain-control both 
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virtual and mechanical actuators while receiving rich visuo-tactile feedback, aimed at restoring autonomous 
locomotion. Common to all periods of the WA-NR was the employment of both: (1) an EEG-based BMI, which 
required patients to produce motor imagery related to walking, was responsible for controlling the initiation of a 
series of lower limb motor behaviors (standing, walking and kicking a soccer ball), and (2) a multi-channel sen- 
sory substitution (remapping) strategy that utilized a haptic display applied to the skin surface of the patients’ 
forearms to deliver both tactile and proprioceptive-like feedback. Such real-time tactile/proprioceptive feedback 
of autonomous bipedal walking was combined with visual feedback (3) during physical training using a robotic 
Body Weight Support (BWS) system on a treadmill (LokomatPro), an overground BWS system (ZeroG), and a 
robotic exoskeleton. 

For all our patients, clinical diagnosis of total (ASIA A) or partial (ASIA B) paralysis was confirmed, over 
multiple years, by routine clinical neurological examination, performed by different neurologists belonging to 
the clinical staff of the hospital in which these patients were followed. Previously, these patients were enrolled ina 
traditional physical rehabilitation program that mainly aimed at increasing independence in daily living activities, 
while seated in a wheelchair. Two patients (P2 and P6) had routine training in a standing orthostatic position 
(stand in table device). Six patients (P1, P3, P4, P5, P6, P7) had walking training using parallel bars or using a 
walker. None of these subjects exhibited any level of sensory or motor improvement or recovery in the many years 
they were followed prior to enrollment in our study. 

At the onset of our protocol, the ASIA status of all eight patients was confirmed by our own initial neurological 
evaluation. That further supports our contention that the neurological improvement observed here resulted only 
from the new WA-NR introduced in the present study. 

Overall, all eight patients involved in the study experienced a significant improvement in tactile, proprio- 
ceptive, vibration, and nociceptive (but not temperature) perception. Such improvement was already noticeable 
after 7 months, but reached its peak at the 10 month of training. On average, such a sensory recovery spanned 
multiple dermatomes below the SCI level, being more vigorous and consistent for altered nociceptive perception 
(more than five dermatomes on average) than for tactile, vibration or proprioception (between one-two dermat- 
omes). Thus, as a rule, the pattern of sensory recovery documented in all eight patients indicated a larger effect 
mediated by small myelinated or non-myelinated fibers, which normally convey nociceptive and high-threshold 
tactile information, than through the large myelinated fibers that normally mediate fine tactile discrimination and 
proprioception. This suggests that axons running through the spinothalamic tract were the main mediators of this 
somatosensory recovery. As such, this observation may imply that the spinothalamic tract may be more resistant 
to the initial SCI and/or remain more amenable than dorsal column-medial lemniscal fibers to underlie plastic 
recovery, even many years after a spinal cord lesion. Interestingly, this is consistent with previous studies in which 
somatosensory plasticity was documented in animals°)*”. 

It is important to mention that, although we have not documented any significant recovery in thermo sensa- 
tion, this negative result may reflect primarily the lack of specificity of the clinical method employed to evaluate 
temperature sensing. In the future, we intend to repeat this analysis using a more sensitive technique. 

In addition to significant sensory recovery, we also observed widespread improvement in voluntary muscle 
control below the level of SCI, even in patients clinically classified as having a complete SCI. Such a recovery in 
motor function, which progressed from proximal to distal muscles over time (Fig. 4A) and was more intense at 
the level of anti-gravitation (extensor) and flexor muscles involved in hip movements — despite the fact that some 
motor recovery was seen at the level of knee and even ankle joints - was corroborated by clinical examination, 
EMG recordings (Fig. 3B,C), and direct measurements of L-force generated by patients (Table 2). Such a pattern 
of motor recovery suggests mediation by intact fibers of the vestibulo-spinal tract (extensor muscles) that run in 
the ventrolateral portion of the spinal cord, next to the spino-thalamic tract. Motor recovery at flexor muscles 
suggests that some fibers of the rubro-spinal tract may have also remained intact in some of our patients. 

Altogether, the partial neurological improvement observed meant that 50% of our patients could be reclassi- 
fied (three from ASIA A to C and one from ASIA B to C) in less than a year of training with our neurorehabilita- 
tion protocol. 

Prior to the present study, the literature contains only a single case report indicating that a patient with tetra- 
plegia was reclassified from ASIA A to ASIA C after 3 years of being subjected to functional electrical stimulation 
bicycle therapy*’. As far as we can tell, no independent study has reproduced this result so far. Heretofore, partial 
neurological recovery after an SCI has been mainly reported in subacute incomplete SCI patients. For instance, 
repetitive transcranial magnetic stimulation (rTMS), applied over the arm and leg representations of the pri- 
mary somatosensory cortex of incomplete SCI patients, led to limited and variable improvements in sensory 
and motor functions***’ primarily when high rTMS intensities were employed. Recently, the use of epidural 
stimulation at the lumbosacral level, combined with standing and stepping training, has allowed chronic ASIA 
A and B SCI patients to voluntary control paralyzed leg muscles. However, such motor control could only occur 
in the presence of the epidural stimulation**. In other words, these patients did not recover the ability to control 
their muscles without the assistive device. As such, none of the four subjects described by Angeli et al.°* changed 
their original ASIA classification. Interestingly, the study methodology also included a pre-implantation phase 
with extensive locomotor training (80 sessions) using a body weight support system on a treadmill. Neurological 
evaluations, neurophysiological measurements and ASIA exams, were performed before and after assisted gait 
training and implantation phases and no significant neurological recovery was observed suggesting that an iso- 
lated gait trainer with a BWS system on a treadmill does not produce meaningful neurological recovery in com- 
plete SCI patients. Moreover, no neurological recovery was described in a recent case report of a single SCI patient 
who was able to walk again using a BMI gait protocol that employed functional electrical stimulation of the lower 
limbs*. 

Since our patients suffered their spinal cord lesions many years before enrolling in our protocol, the likelihood 
that the sensorimotor improvements observed here were due to spontaneous recovery can be basically ruled out. 
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Figure 6. Hypothesis for mechanism of neurological improvement in SCI patients. (A) Example of spinal 
cord lesion in the thoracic area. Cross section is shown in the three parts of the spinal cord: at the lesion level, 
on top and under the lesion. Example of two efferent and one afferent pathways and their corresponding spinal 
tract are shown (rubrospinal and vestibulospinal tracts; spinothalamic tract respectively). At the lesion level 
we hypothesize preservation of 2-25% of white matter’* which might include the spinothalamic tract (sensory: 
pain, temperature, crude touch and pressure), vestibulospinal tract (motor: extensors muscles), rubrospinal 
tract (motor: flexors muscles) and dorsal columns (sensory: proprioception and fine touch). Under the lesion, 
Central Pattern Generators (CPGs) and its interaction with descending pathways (reticulospinal tract) and 
sensory afferents, modulating the gait pattern®’. (B) Proposed components for the rehabilitation mechanism: 
direct brain control of virtual or robotic legs, continuous stream of tactile stimulation representing the missing 
haptic feedback from the legs and robotic actuators to train patients to walk. Cortical and spinal plasticity are 
hypothesized to change and to modulate neurological circuits in the preserved area around the lesion through 
motor (red) and sensory (blue) connections. 


Indeed, a review by the International Campaign for Cures of Spinal Cord Injury Paralysis® about spontaneous 
recovery after SCI, based on pharmaceutical clinical trials that focused on acute neuroprotection in SCI®-°, 
reported that the majority of spontaneous recovery occurs during the first 3 months after the SCI. Small residual 
clinical improvements can persist for up to 18 months, but only minor changes occur afterwards. Thus, 1 year 
after an SCI, 80% of the initial ASIA A cases remain A, about 10% convert to ASIA B and about 10% to ASIA C. 
A survey of 987 SCI patients showed that, between 1 and 5 years after the lesion, a conversion from ASIA A toa 
higher grade occurs in only 6.5% of patients (3.5% to B, 1.05% to C and 1.05% to D)®*. Since some of the patients 
that moved from ASIA A to ASIA C in the present study had suffered their SCI more than a decade ago, it is highly 
unlikely that spontaneous recovery accounts for our findings. 

In complete motor lesions (ASIA A and B), the majority of functional recovery occurs within the ZPP, follow- 
ing a craniocaudal sequence. Recovery within the ZPP appears to be due to both CNS and peripheral plasticity, 
while recovery beyond the ZPP would probably demand some CNS repair, likely involving axon regeneration”. 
Concerning partial motor recovery, the same review indicates that it likely occurs in myotomes with sensory 
preservation. Overall, the chances of a recovery of more than two spinal levels below the initial ASIA level are 
very small. Our findings revealed that motor recovery was indeed more significant within the ZPP. However, we 
also observed patients’ partial recovery in voluntary motor activity located in more than two dermatomes below 
the ZPP. 

A previous study with ASIA B patients™ suggested that preservation of pinprick sensation can be useful in 
predicting motor recovery: presence of sacral pinprick sensation 4 weeks after the SCI significantly predicted 
ambulation 1 year later. Although these findings are not directly comparable to ours, by the differences in patient 
samples, the fact that we were able to document a 50% improvement in ASIA classification after many years of SCI 
raises the hypothesis that further motor clinical improvement could be seen with longer training. 

The clear functional significance of the sensorimotor recovery observed here was further demonstrated by 
both the major overall improvement in the patients’ Walking Index. In other words, the observed partial sensori- 
motor recovery was translated into a meaningful improvement in the patients’ daily routine. 

But what mechanism could account for this partial neurological recovery? Kakulas et al.!® showed that about 
60% of SCI patients diagnosed clinically as having a complete spinal cord injury (ASIA A) still have 2-27% of 
the total area of spinal cord white matter preserved"*. This finding is further supported by the observation that 
some of these surviving axons can exhibit functionality’” in more than 80% of such ASIA A patients. Sherwood 
and colleagues defined these cases as having a “discomplete” SCI and suggested that the residual axons could be 
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functionally rescued to mediate some level of clinical recovery!’. We propose that such a mechanism may have 
accounted for the partial neurological recovery observed in our study (Fig. 6A). 

Assuming that residual spinal cord connectivity mediated the clinical recovery, what are the potential phys- 
iological mechanisms involved? Concomitantly to the partial neurological recovery, we documented the occur- 
rence of cortical functional plasticity through longitudinal analysis of EEG recordings. This plasticity manifested 
itself by the emergence of consistent activation in the leg representation area of S1/M1, as measured by ICA, 
event-related spectrogram perturbation, and event-related potential analysis. The most prevalent EEG feature 
identified over training was the desynchronization of mu rhythm (7.5-12.5 Hz)*!*. Secondarily, we have also 
observed an increase in beta wave desynchronization. 

Altogether, these results are consistent with our prior hypothesis®* that long-term BMI use is capable of 
changing the cortical body representation by including either new artificial actuators (robotic limbs or avatar 
bodies) or even by inducing a reactivation of the representation for paralyzed limbs, in the present case, legs. 
Therefore, based on these EEG findings, we propose that our long-term BMI-based training triggered a signif- 
icant process of functional plasticity in $1/M1. Such functional cortical plasticity may have accounted for the 
re-emergence of lower limb representations in these cortical areas, as documented by the EEG analysis described 
above, and in another study with the same patients*’. Such a functional cortical plasticity likely led to the reacti- 
vation of upper motor cortical neurons that normally project to the spinal cord, via the corticospinal tract. Given 
that a small fraction of spinothalamic, vestibulospinal and rubrospinal tract axons may have survived the initial 
SCI event and remained silent for many years, even in our ASIA A patients, the peculiar motor recovery observed 
in our study, involving primarily hip extensor and flexor muscles, could be explained by the functional reacti- 
vation of these residual axons as a byproduct of plasticity induced by long-term, intensive BMI training (Fig. 5). 

But what are the key components of BMI training that triggered such a massive cortical plasticity? In our 
view, the driving force behind this plasticity includes: the direct brain control of robotic actuators by attentive 
and motivated patients, the reliance on patient's motor imagery of walking to operate a BMI, a continuous stream 
of rich tactile/proprioceptive feedback, and the use of robotic actuators that allowed patients to routinely walk 
upright for long periods of time. This last feature may also have accounted for the generation of complex interac- 
tions between the supraspinal centers (cortical and subcortical structures) and the spinal cord. 

Particularly, in the case of locomotion, the use of robotic gait training may have triggered the engagement of 
central pattern generators (CP), both at the supraspinal and spinal levels, and contributed to the generation of 
tactile and proprioceptive feedback from the patients’ own legs. These two components may have also induced 
functional plasticity at the spinal level and contributed to the type of sensory recovery observed below the level of 
the SCI, mediated mainly by the spinothalamic tract, and the somatosensory cortical plasticity described above. 

In both animals and humans”, central pattern generators (CPGs) have been shown to generate bilateral 
rhythmic patterns, alternating motor activity between the flexor and extensor motoneurons in the absence of 
descending inputs’!. Animal studies have shown that the gait pattern itself is generated in the spinal cord by 
CPGs, which are modulated by a peripheral sensory feedback. Conversely, once gait is re-established, proprio- 
ceptive and cutaneous afferents signals derived from receptors located in muscles, joints, and skin, can once again 
drive intraspinal circuits that interact with motor neurons, interneurons and CPGs in order to assist movement 
adaptations, such as postural corrections®. The level of this CPG activity would be determined by the brainstem 
locomotor command systems through the reticulospinal pathways® (Fig. 5). The existence of CPGs in humans 
has been suggested, but its exact location is still unclear’. 

Another possible source for sensory improvement observed in our patients could be the long-term use of 
visuo-tactile stimulation in virtual reality. Sensory modalities are not independent from each other; experiments 
have shown mechanisms of cross modal interaction’, and cross modal integration to create a robust percep- 
tion”*. In particular, vision of a body part was found to influence tactile perception’*’””. For example, one experi- 
ment has shown that observation of a body part increases tactile acuity during passive touch in healthy subjects”. 
‘The effect was present when subjects were touched on their forearm while observing a different part of their 
own arm; and their accuracy was even increased when the body part was magnified”. No effect was observed 
if a neutral object was used in place of the arm. Patients with sensory deficiency (stroke patients) were found to 
improve their sensory resolution by observing their own body. In another experiment, magnifying the arm of a 
patient with chronic pain increased his/her pain, while minifying decreased it®°. Similarly, we hypothesize that 
long-term training observing a human avatar mimicking the position and orientation of the patients’ body could 
have induced a positive effect on our patients’ sensory acuity. 

Based on our clinical findings, we propose that long-term gait training with a BWS that employs BMI-based 
robotic actuators, combined with rich tactile feedback, could recruit the activation of CPGs in SCI patients*!. 
Likely, BMI-based training and tactile feedback in a virtual reality environment could also enhance CPG activity 
by recruiting cortical afferents to influence locomotion control. If some corticospinal or vestibulospinal axons are 
still intact in a fraction of SCI patients, these locomotion-related signals could reach lower alpha-motor neurons 
below the level of the SCI. Moreover, by making patients walk routinely upright and against load, peripheral tac- 
tile and proprioceptive feedback would be generated and transmitted back to the spinal cord, contributing to the 
process of spinal cord functional reorganization (Fig. 6B). 

In addition to partial neurological recovery, we observed a clear linear correlation between the number of 
hours spent upright walking/standing with the amount of bowel function (Fig. 4E). These and other autonomic 
improvements will be further explored in future studies using the same methodology. 

We observed that significant clinical recovery was closely related to long-term and frequent use of a BMI par- 
adigm that attempts to recreate lower limb movements in a realistic way (either in a virtual reality environment 
or through the use of brain-controlled robotic walkers). Further support for this contention can be found in the 
observation that after the two 30-day periods of vacation afforded to patients during the 12 month duration of 
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the protocol, we noticed a reduction in sensory and motor capabilities below the SCI level (Figs 2 and 3). Such a 
temporary reduction was quickly reversed, however, by the restart of the BMI-based training protocol. 

Introduction of rich tactile feedback signals, delivered via a haptic display, also seemed to have played a key 
role in the patient’s recovery. Experiments reported elsewhere have shown that patients are capable of incorpo- 
rating an avatar body and extending their body schema toward the avatar legs employed in our training”. This 
suggests that the introduction of tactile feedback likely enhanced the ability of patients to exhibit cortical and/ 
or subcortical functional plasticity during training with our BMI protocol, as we had previously documented in 
monkeys®”*, 

Overall, the results obtained in our study suggest that BMI applications should be upgraded from merely a 
new type of assistive technology to help patients regain mobility, through the use of brain-controlled prosthetic 
devices, to a potentially new neurorehabilitation therapy, capable of inducing partial recovery of key neurological 
functions. Such a clinical potential was not anticipated by original BMI studies. Therefore, the present findings 
raise the relevance of BMI-based paradigms, regarding their impact on SCI patient rehabilitation. In this context, 
it would be very interesting to repeat the present study using a population of patients who suffered a SCI just a 
few months prior to the initiation of BMI training. We intend to pursue this line of inquiry next. Based on our 
findings, we anticipate that this population may exhibit even better levels of partial neurological recovery through 
the employment of our BMI protocol. 
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Spinal cord injuries disrupt bidirectional communication between the patient's brain and body. Here, 
we demonstrate a new approach for reproducing lower limb somatosensory feedback in paraplegics 
by remapping missing leg/foot tactile sensations onto the skin of patients’ forearms. A portable haptic 
display was tested in eight patients in a setup where the lower limbs were simulated using immersive 
virtual reality (VR). For six out of eight patients, the haptic display induced the realistic illusion of 
walking on three different types of floor surfaces: beach sand, a paved street or grass. Additionally, 

: patients experienced the movements of the virtual legs during the swing phase or the sensation of the 

: foot rolling on the floor while walking. Relying solely on this tactile feedback, patients reported the 

: position of the avatar leg during virtual walking. Crossmodal interference between vision of the virtual 
legs and tactile feedback revealed that patients assimilated the virtual lower limbs as if they were their 
own legs. We propose that the addition of tactile feedback to neuroprosthetic devices is essential to 
restore a full lower limb perceptual experience in spinal cord injury (SCI) patients, and will ultimately, 
lead to a higher rate of prosthetic acceptance/use and a better level of motor proficiency. 


Spinal cord injuries can induce significant bidirectional loss of communication between the subject’s brain and 
his/her body, i.e. the patient can neither generate body movements as a result of the loss of cortical communica- 
tion with the spinal cord, nor can their brain receive somatosensory feedback originating in the body’s periphery’. 
For the past 15 years, studies in animals”? and patients* have suggested that brain-machine interfaces (BMI) may 
provide a novel therapeutic alternative to restore mobility in severely paralyzed patients®. Yet, despite the high 
prevalence of concurrent somatosensory and motor deficits in such patients, few BMI studies have aimed at 
restoring both motor and somatosensory feedback simultaneously in paralyzed subjects*®. 

In the absence of tactile feedback, BMI users have to rely mainly on vision to enact their direct brain control of 
an artificial actuator. Since in these cases the user does not receive any tactile or proprioceptive feedback related 
to the actuator performance, we and others have argued that such BMI setups are unlikely to have a significant 
clinical impact on severely impaired patients*. This is easy to understand when one realizes that a patient would 
have to look at his prosthetic hand every time he moves it to reach and hold an object, or look at the floor every 
time his lower limb exoskeleton touches the ground during autonomous locomotion. 

As part of a project to develop and test a non-invasive, brain-controlled lower body exoskeleton for SCI 
patients, we implemented a new rehabilitation paradigm in which patients interacted with a rich immersive vir- 
tual reality environment’ (VRE). In this VRE, patients interacted with the movements of 3D virtual avatar legs on 
different surfaces and environments, through a virtual reality head mounted display (Fig. 1a). During this task, 
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Figure 1. Tactile shirt and pseudo proprioception results. (a) Subject wearing the tactile shirt. Three 
vibrators are aligned on ulna bone. Detachable vibrators are connected to a PCB converted by cloth and Velcro 
bands. Details of the vibrator modules are shown: the clipping systems are used both as mechanical clipping 
and electrical connectors (power + data). (b) Two different tactile paradigms are proposed: feedback on stance 
(tactile feedback rolls on the forearm of the patient in synchrony with the foot contact) or swing (feedback roll 
during the swing and all three vibrators at once on foot contact). (c) Results given for the experiment evaluating 
the pseudo proprioception test with four different tactile feedbacks (left to right): 1) feedback on stance moving 
from wrist to elbow (Distal to Proximal, DtP), 2) feedback on stance moving from elbow to wrist (Proximal to 
Distal, PtD), 3) on swing DtP and 4) on swing PtD. Patients used their arms to show the position of the walking 
avatar’s legs, relying on tactile feedback only. Scores based on ability to reproduce the avatar walk for different 
speeds are reported including the number of patients for whom a certain paradigm was the best or the worst 
choice and the rank of the four paradigms. 


patients also wore a haptic display device integrated in the long sleeves of a shirt (Fig. 1a). Each of the sleeves con- 
tained three small coin-shaped vibrators lined up along the forearm’s distal-proximal axis. By varying the magni- 
tude and temporal sequence of activation of these vibrators, we were able to deliver somatosensory information, 
originating from the movements of virtual limbs observed by the eight SCI patients, onto the skin surface of their 
forearms.. Our central goal was to allow these SCI patients to use this haptic display in order to sense the position 
of the virtual legs in space, the contact of the virtual foot with the floor, and the type of surface with which they 
were in contact. In this context, the tactile paradigm presented here is at the cross-roads of two major concepts: 
sensory substitution and sensory remapping. Sensory substitution consists of using one sensory modality to carry 
information from another one’. This can be seen in the classic studies of Bach-y-Rita in which visually impaired 
patients learned to use a matrix of vibrators to translate visual information into tactile feedback delivered on the 
skin of the back® or on the tongue’s surface’. Haptic displays have also been used to mitigate audition’ or vestib- 
ular deficits''!? among others. On the other hand, sensory remapping techniques (for example those used with 
amputee patients) aim at providing somatotopically matching haptic stimulation onto a different body part". 
Here, we implemented a paradigm that includes: (a) a non-invasive tactile remapping, obtained by provid- 
ing vibrotactile'* stimulation on the patient’s forearms to inform about lower limb tactile and proprioceptive 
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information during a virtual gait/walking task; and (b) a tactile substitution procedure in which complex hap- 
tic cues (tactile and proprioceptive information from lower limbs) were replaced with simpler encoded tactile 
stimuli. 

Using this apparatus, SCI patients were able to receive both the tactile feedback providing temporal and physi- 
cal cues about virtual feet interacting with different floor surfaces, and the proprioceptive feedback describing the 
position in space of virtual lower limbs. Analysis of the patients’ performance in a crossmodal congruency task’* 
revealed that subjects experienced an assimilation of the virtual legs as a projection of their own body. 


Results 

All experiments were performed with naive SCI patients (Table $1) using an immersive virtual reality system 
where the subjects’ lower limbs were simulated by a human-like 3D avatar seen from the first person’s perspec- 
tive through a head mounted display (Fig. 1a). Tactile and proprioceptive sensations generated by the avatar’s 
virtual legs were mapped on the patients’ forearms by means of arrays of vibrators that defined a haptic display 
(Fig. 1a). Initially, we were interested in identifying the most intuitive tactile feedback paradigm for SCI patients 
to perceive the position of their legs when relying on tactile feedback only. Next, we analyzed how the avatar legs 
were assimilated into the patients’ brain body representation using the same apparatus. Lastly, we investigated the 
parameters of tactile stimulation that allowed our patients to experience the vivid sensation of walking on three 
different types of ground surface: sand, grass or paved street. 


An Intuitive Representation of Leg Position Using a Forearm Haptic Display. We ran experiments 
with seven (P1-P7) patients. Here, we considered different tactile feedback modalities and searched for the one 
that patients felt was more intuitive for perceiving the position of the avatar leg when relying on the tactile feed- 
back solely. Two main questions were asked: (a) Should the artificial feedback represent the rolling of the foot on 
the floor (feedback on stance, Fig. 1b) or the swinging of the leg when the body is balancing; and (b) Should the 
tactile stimulation be delivered from the wrist to the elbow (Distal to Proximal, DtP) or from the elbow to the 
wrist (Proximal to Distal, PtD). This was important as the artificial tactile feedback proposed here is an abstract 
substitution of the sensory feedback from the legs. Thus, the direction of the movement on the forearm can be 
perceived as representing different phases of the gait cycle. For example, during the stance phase, a PtD sequence 
of forearm vibration reproduces the same direction of movement produced by the foot rolling on the floor (i.e. 
perception of the foot going forward). Meanwhile, a DtP sequence moves in the same direction as the foot’s move- 
ment in relation to the subject’s body (perception of the body going forward compared to the foot during stance). 

To induce a movement sensation on the forearm while using an array of eccentric rotating mass (ERM) vibra- 
tors, we employed a well-documented illusion called the tactile apparent movement'*””, This illusion appears 
when two vibrating actuators are sequentially triggered on the subject’s skin with a specific stimulation length 
and specific delay between the onset of the two vibrations. As a result, the subject perceives one continuous touch 
going from the first to second vibrator instead of two discrete stimuli. The parameters of stimulation length 
and onset delay inducing the apparent movement illusion were calculated for each patient (see Supplementary 
Methods, Fig. S1). 

During the experiment, patients were asked to report the position of the avatar leg relying on tactile feedback 
only. The avatar’s walk was randomly changed between three speeds (Fig. $2a). A score, ranging from 0 to 3, was 
given to evaluate a patient’s performance for each speed: 0 - if the patient could not report the position of the 
avatar leg; 1 - if patients could report the position at constant speed; 2 - if they could report the position when the 
speed was changed, but confused the left and right leg; and 3 - for a perfect execution of the task. Final score was 
the sum of scores for the three speeds and thus ranged between 0 and 9. 

Three out of seven patients accurately reported the leg position at every speed when the tactile feedback was 
of type Stance-Distal to Proximal (DtP), three had good performance (score range 5-8), and one was unable to 
report the leg position (Fig. 1c, Table S2 for details). For the Stance-Proximal to Distal (PtD) paradigm, three 
patients had good or perfect performances whereas the other four had average to bad performance (score range 
1-4). Three patients were perfect with the Swing-Distal to Proximal (DtP) tactile paradigm, two were good and 
the last two were average or bad. Finally, for Swing-Proximal to Distal (PtD) paradigm, four patients exhib- 
ited average performance, one was good and two were perfect. We observed that four patients had their highest 
score with Stance-DtP, one reached maximal performance with Stance-PtD, three with Swing-DtP and two with 
Swing-PtD. 

Not every patient had his/her best performance with the same tactile paradigm. However, we wanted to use 
only one type of tactile feedback for the whole group for all tests with the tactile shirt. Therefore, we identi- 
fied the type of feedback that elicited the best performance in a majority of our subjects. Using a simple metric 
ranking (see Methods), we classified the four paradigms as follows (from best to worst): Stance-DtP, Swing-DtP, 
Swing-PtD and Stance-PtD (Fig. 1c). Overall, tactile stimulation going from Distal to Proximal was more intuitive 
for feedback delivered during both the avatar leg Stance and Swing phases. At the end of the experiment, patients 
were also asked to report their subjective experience (Fig. S2b). All patients answered that they ‘agreed’ or ‘totally 
agreed’ that ‘the tactile feedback [matched] the walk’ when the tactile feedback was on Stance-DtP, Swing-DtP, 
and Swing-PtD. For Stance-PtD, one patient ‘disagreed’ that feedback matched the walk. No patient reported 
that “The tactile feedback was disturbing’ for any of the four feedback modialities. All patients ‘agreed’ or ‘totally 
agreed’ they ‘could imagine/feel [themselves] walking well with [their] eyes closed, and [they] knew where the 
avatar legs were’ for Stance-DtP, Swing-DtP, and Swing-PtD, while one patient disagreed with that statement 
during the Stance-PtD modality (Fig. S2b). 

Considering both the behavioral assessment and the responses to the questionnaire, it appeared that feedback 
on Stance, with a sequence of vibrations going from wrist to elbow (DtP) was the best feedback at the group level 
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Figure 2. Crossmodal congruency task. (a) Experimental setup, subject wore the tactile shirt and the head 
mounted display. Vibrators were placed on patients’ forearms and visual distractors on the virtual avatar’s toe 
and heel. (b) A cross congruent task (CCT) block is run after a Vision only block (VO: 1 minute observation of 
the avatar walking) or a VT block (same as VO with tactile feedback on the stance). After a random sequence of 
2 VO and 2 VT blocks, we ended the session with another sequence of CCT block following a VO or a VT block 
(c) Mean and standard error for the Cross Congruent Effect (CCE). Yellow bar shows average response time 

for trials where tactile stimulation and visual distractors were on the same arm and pink bar shows those where 
they were contralateral. P values for T-test are shown. 


to deliver tactile feedback to our patients using our forearm haptic display. Note that during the experiments 
patients were never told what the vibrotactile pattern represented (if it was on Stance or on Swing, etc.). This 
allowed them to make bias-free connections between the vibrotactile stimulation and the avatar’s walk. 


Incorporation of Virtual Legs with Visuo-Tactile Stimulation. After we identified the most intuitive 
protocol for delivering tactile feedback, we investigated the change in the patients’ perception of the avatar legs in 
relation to their own bodies while using our haptic display. We used a psychophysical measurement while tactile 
feedback (of type Stance-DtP) was synchronized with the avatar walk (Vision + Tactile, VT). We compared this 
condition to the case where tactile feedback was absent and only visual feedback was provided (Vision only, VO). 

We employed the Crossmodal Congruency Task (CCT)!*!*"9, a well-documented indirect measurement of 
tool incorporation by human subjects, and the corresponding Cross Congruent Effect (CCE), which calculates 
the interaction between a vibration on the body and a visual distractor on the tip of the tool (in our case the avatar 
feet), to measure how the virtual avatar legs were assimilated by our SCI patients (high CCE correlates with inte- 
gration of the tool; see Experimental Procedures). All our patients were able to discriminate the position of the 
vibrators used in this experiment (Fig. S3). 

For all patients, at the beginning of the session, we ran a CCT block succeeding a first Vision Only block 
(VO,) and a CCT block succeeding a Vision + Tactile block (VT,) (Fig. 2a,b). CCE for the same side and CCE 
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_Congruent | _Incongruent | _ Mean CCE 
YO Diff side 787.1+18.6 828.1419.9 41.0+8.0 
Same side 755.4+20.8 789.3 +19.7 33.9+8.3 
Beginning of the session 
a Diff side 807.8 + 18.4 804.4 + 20.6 —3.4+8.1 
Same side 748.1417.2 855.8 + 24.4 107.6+8.4 
Toa Diff side 820.7 + 22.2 855.1+22.0 34.44+9.1 
Same side 762.44 15.5 829.7421.2 67 447.3 
End of the session 
om Diff side 815.9+24.5 837.8+21.9 21.9+9.0 
Same side 792.0+ 19.8 861.5+23.4 69.5+8.3 


Table 1. Mean Response Cross Congruent Effect over all patients. Mean + standard error of reaction 

time (in ms) over all eight patients for blocks succeeding a Vision Only (VO) block or blocks succeeding a 
Vision + Tactile (VT) block. Mean CCE corresponds to the mean RT for congruent - mean RT for incongruent 
visuo-tactile simulation is show in the right- most columns. 


for different sides refer to the cases where the visual distractors and the vibration are on the same arm or on the 
contralateral arm. CCE(M,) refers to the Cross Congruent effect recorded right after the n block of a modality 
M. The mean reaction times (RTs), for all conditions over all patients, are shown in Table 1. 

Mean CCE(VO,), calculated as the RT difference between trials where the tactile stimulation on the forearm 
and the visual stimulation on the avatar foot had incongruent elevations (e.g. vibration on the proximal vibrator 
and visual stimuli on the tip of the foot) and trials where they were congruent, was equal to 41 +8 ms (+SEM) 
when the vibrator and distractor were on a different side (incongruent: 828 + 20[ms], congruent:787 + 19 [ms]) 
and 33 + 8 ms for the same side (incon: 755 + 21 [ms], cong: 787 + 18 [ms]). 

We found that CCE(VT,) was —3 + 8 ms for different sides (incon: 804 + 21 [ms], cong: 807 + 18 [ms]) versus 
107 + 8 ms for same side (incon: 855 + 24 [ms], cong: 748 + 17 [ms]). Three-way analysis of variance (ANOVA) 
with the factor congruency (congruent/incongruent), side (same side/different side) and modality (VT/VO) on 
RT, showed a significant effect of congruency (P < 0.01), and a significant effect of the interaction between side 
and congruency (P < 0.05). CCE was higher for same side than for different side delivery of distractor and tactile 
stimulation. We also found a three-way interaction between congruency, side and modality (P < 0.05). A post-hoc 
analysis with Bonferroni correction was run to determine the difference between CCE(VT,) and CCE(VO,). CCE 
was significantly higher for same side compared to different sides (P < 0.01) for VT modality. No difference was 
found for CCE(VO,) between the same and different sides (P > 0.5) (Fig. 2c). In other words, we found a signifi- 
cant CCE effect (used as the indirect measurement of the avatar leg assimilation) after the first VT block and no 
such effect was observed right after the first VO block. 

Interestingly, when we repeated two more blocks of VO and VT (VO), VO; and VT,, VT; randomly sequenced, 
Fig. 2b), and then ran CCE(VO,) and CCE(VT,), we observed different dynamics (see RTs in Table 1). The mean 
RT difference of CCE(VO,) was 34.4+ 9.1 ms for different sides and 67.4 + 7.3 ms for the same side; CCE(VT,) 
was 21.9+9.0 ms for different sides and 69.5 + 8.301 ms for the same side. Significant differences between the 
same side and different sides were found both in CCE(VT,) (P < 0.001) and CCE(VO,) (P < 0.05). Simply stated, 
there was a significant increase in RT during the CCE applied after a VO block at the end of the session; some- 
thing that did not happen at the beginning of the session. 

Altogether, these results suggest that patients extended their body representation to integrate the avatar legs 
after using the visuo-tactile stimulation. 


Simulation of Floor Texture. Patients explored and selected 120 times per session the set of tactile param- 
eters that best represented for each of them the sensation of walking on three virtual ground textures labeled as: 
sand (SAT), grass (GRT) and paved street (PST) (exploratory phase). In other words, while seeing through the 
head mounted display, a black control surface or the images of three distinct ground surfaces (sand, grass, and 
pavement) upon which the avatar was walking, patients received a variety of tactile feedback patterns on their 
forearms. Four factors of the tactile feedback were varied to create a catalog of perceived tactile sensations: ampli- 
tude of the distal, middle and proximal vibrators (DV, MV and PV) and the stimulation timing (ST) (Fig. 3a—d). 
See Floor Texture Simulation Test section in Methods for a description of the procedure to explore the full catalog 
of sensations presented to the patient. This exploratory session lasted for 40-45 minutes. At the end of the session, 
a complete catalog correlating tactile stimulation patterns and ground surfaces was obtained for each patient. 

After a 20-minute break, we removed the floor from the virtual reality simulation and begin to replay the set of 
chosen tactile stimulation parameters in a random order, using the same haptic display applied to the skin of the 
patients’ forearm. Now we asked the same patients to report the floor type to which each set of tactile stimulation 
parameters corresponded (inverse task phase). Figure 4a shows that in 11 out of 15 sessions, the patients’ accu- 
racy rate was significantly above chance (mean + 20 = 0.42). In other words, most patients were able to correctly 
describe the ground texture originally associated with a particular set of tactile parameters used to stimulate their 
forearm skin. Overall, 6 out of 8 patients performed this task above chance: patients P2, P3, P4, P6, and P7 were 
significantly above chance in both of their sessions; patient P5 was significantly better than chance in one of two 
sessions. Details of the patients’ performance per floor type and session are shown in Fig. 4a. Mean accuracy was 
significantly higher than chance for all three surfaces (Fig. 4b). However, patients were better at finding SAT than 
GRT (multiple comparisons, Bonferroni correction, N= 14, P< 0.01) and PST (P < 0.05). 
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Figure 3. Floor texture simulation, setup. (a) Subjects wore the tactile shirt and the head mounted display; a 
3D avatar was walking on either an empty floor or one of the three ground textures: Sand (SAT), Grass (GRT), 
and Paved Street (PST). (b) A tracking system detected the coordinates of the hand position and translated 

to a tactile modality defined by the amplitude of Proximal, Middle and Distal Vibrators (PV,MV,DV) and the 
Stimulation Timing (ST). (c) The axis of the four parameters describing the tactile feedback was randomized 

at each trial. Example of axis configuration and corresponding choices for SAT given by a subject. (d) Varying 
the Stimulation Timing (1 to 10) changes the Inter-Stimulus Onset Interval (ISOI) and Duration of Stimulation 
(DoS) parameters of the tactile feedback. It is possible to simulate the following sensation of touch: one single 
vibration on the whole forearm, a continuous feedback going from distal to proximal vibrator positions or three 
distinct vibrations on distal then middle and proximal forearm. 
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Figure 4. Floor texture simulation, results. (a) During the inverse task the parameters chosen during the 
exploratory phase were replayed in a random order; subject had to report the floor type relying on the tactile 
feedback only (chance level = 0.42). (b) Patients’ accuracy per floor type (multiple comparisons, N = 14) 

and for trials with or without floor. (c) Mean answer over all patients and responses to Q1 (asked prior to the 
experiment): ‘I remember the sensation of my feet on X’ (X: SAT,GRT or PST, ‘I strongly disagree’ —2 to ‘T 
strongly agree’ +2). (d) Detail of patients responses for Q1. (e) Subjects’ responses to question ‘I had during 
the experiment the sensation of walking on X’ after a V only session (patients observed the avatar walking) 
and a Vision + tactile session (tactile feedback on stance) (*for P < 0.05, Wilcoxon test). (f) Detail of patients’ 
answers to question Q2. (g) Principal tactile factors of the experiment, ground types are classified (using 
knn classification) using each one of the four factors. Mean classification accuracy over all sessions for four 
factors (multiple-comparison test, N = 14, *: P< 0.05). (h) Mean and standard deviation for principal factors: 
Amplitude of proximal Vibration (PV) and Stimulation Timing (ST) for all sessions. Color convention as 
follows: red for SAT, green for GRT and blue for PST. (i) Values from panel h are joined following structure 
similarities (e.g. sessions where SAT was on top left corner and PST on the lower right). 


During the exploratory phase, in half of the trials, patients could see the avatar walking on a virtual ground 
that mimicked the one they were asked by the experimenter to identify (Fig. 3a), while in the other half of the 
trials no ground surface was visible under the avatar’s feet (no floor). No difference was found between trials 
in which the ground was shown and trials where the ground was absent from the virtual environment (T-test, 
P>0.5) (Fig. 4b). 

While all patients suffered a spinal cord lesion at least 3 years (and some as much as 10 years) prior to the onset 
of the experiment (Table S1), all patients except P8 reported still having vivid memories of the sensation produced 
by the interaction of their feet with sand and grass (Fig. 4c,d). For the sensation of walking on a paved street, 
all patients - except P5 and P8 - also reported such memories. After the experiment, six out of eight patients 
reported that they experienced again the vivid sensation of walking on the three chosen ground surfaces (Fig. 4e,f 
‘Vision + Tactile’). P5 and P8 were the only patients that did not report this sensation. Patients’ responses were 
translated into numerical grades (Fig. 4e, —2 for ‘I fully disagree’ to 2 for ‘I fully agree’). The group average (N=8) 
was positive for all three conditions: 0.88 for SAT and GRT, 0.75 for PST. 

The patients’ responses were different if the same question was asked after a session where no tactile feedback 
was given (Fig. 4e,f Vision Only). In this control experiment, patients only observed the avatar walking, but did 
not receive tactile feedback. Patients’ answers were close to 0 for all three conditions when asked if they had the 
feeling of walking on the ground surfaces: paved street (0.25), grass (—0.13) and sand (—0.13). 

Differences between “Vision Only” and “Vision + Tactile” were significant for SAT and GRT (Fig. 4e, 
Wilcoxon test, P< 0.05, N =8), but not for the PST. At the end of “Vision Only” sessions, patients were also asked 
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Table 2. ANOVA on Euclidian distance between chosen parameters for floor textures. For each pair of 
texture i and j we compared the distribution of the Euclidian distances of all combinations of trials for both 
textures to the distribution of Euclidian distance of trials of type i and trials of type j. We reported stars in the 
cell only when the distance between the textures was significantly bigger than the distance between trials for 
each texture taken individually (*for P < 0.05, **for P < 0.01, ***for P <0.001). 


to report on the visual realism of the different ground surfaces. No significant difference was found among the 
three surfaces (P = 0.26, ANOVA). The ‘sensation of walking on X’ after a vision only session was not correlated 
with the patients’ report of the visual realism of the ground surface (Fig. $4b) (R? < 0.01), suggesting that the 
vision of the avatar and the interaction of the avatar with the ground are more important than the visual realism 
of the ground itself. 

Next, we analyzed the principal factors that patients relied on to differentiate the ground type (see 
Supplementary Methods). The stimulation timing (ST) was found to be the primary factor (Fig. 4g and S4c for 
details per patients) while the amplitude of the proximal vibrator (PV) came in second. Both were better than the 
middle (MD) and distal vibrator (DV, multiple comparisons, N = 14, P < 0.01). This suggests that patients relied 
on the tactile vibration level produced at the end of the avatar leg stance phase or the moment in which the foot 
pushed off the floor to move to a swing movement. Surprisingly, the patients seemed not to consider the vibration 
amplitude at the beginning of the stance (when the heel strikes the floor) as an important feature for discriminat- 
ing the ground texture. 

We used these two main factors (PV and ST) to characterize the way patients chose each ground type. 
Figure 4h shows the parameters’ mean and standard deviation for all three surfaces. Similarities can be seen in 
patients 3, 5, 6, 7 and 8 (shown in Fig. 4i). For SAT, these patients expect three distinct stimuli during stance with 
a light feedback when the foot pushes off the ground. On the contrary, for paved street PST they were looking for 
one long stimulation with a strong proximal vibration. GRT was somehow in between sand and paved street, with 
a middle range vibration amplitude for PV. The probability of these similarities being due to chance was found to 
be very low (P < 0.001, see Supplementary Methods). 

Parameters chosen by the patients to describe the ground types were significantly different from each other 
for at least two textures in 10 out of 15 sessions (Table 2 and Supplementary Methods). The most common distin- 
guishable textures were sand and paved street. No difference was found between trials where the floor was shown 
in the VR and those where it was not. Thus, the parameters selected by the patients to identify a given ground 
surface were not influenced by the presence of that ground in the virtual reality environment. 

We also noticed that sessions where patients chose the same factors for all textures during the exploratory 
phase, namely Patient 1, Sessions 1 and 2; Patient 5, Session 2; and Patient 8, Session 1, correspond to the sessions 
where the subjects were not able to differentiate the floor textures during the inverse task. P1 was the patient 
with the lowest score in the sensory discrimination task (Fig. S3), and also the one with the lowest score in the 
Stance-DtP paradigm during the pseudo-proprioceptive test. 


Discussion 
A novel solution to overcome sensory deficiency in the lower limbs for patients with Spinal Cord Injury (SCI) is 
proposed. In this study, missing haptic sensation from the lower limbs was replaced by rich tactile stimulation on 
the skin of SCI patients’ forearms. This feedback was integrated with an immersive virtual reality environment 
where a 3D human avatar was simulated. Three major effects of integration of the tactile feedback with the simu- 
lated 3D avatar were observed: (a) SCI patients could rely on this feedback to perceive the position of the virtual 
leg during locomotion; (b) patients incorporated the avatar legs as an extension of their body schema; and (c) 
patients experienced a realistic sensation of walking on different ground surfaces relying on the tactile feedback 
only. 

Interestingly, patients were never instructed on what the tactile feedback displayed or represented. As totally 
naive subjects, they acquired vivid tactile/proprioceptive sensations after exposure to 1 minute of synchronous 
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visuo-tactile stimulation (patients observed the 3D human avatar walking and received the tactile stimulation on 
their forearm). This suggests that such tactile feedback became intuitive to the subjects rather quickly. As such, we 
propose that our haptic display paradigm was capable of inducing patients to experience a proprioceptive illusion 
that allowed them to deduce the position of the virtual avatar legs relying solely on the tactile feedback. 

Human perception is multimodal” and different sensory inputs are merged and weighted in a statistically 
optimal fashion in order to reduce the variance of the final estimate”!. Each sensory input has proficiencies: 
vision can provide precise spatial encoding, audition is more likely to accurately convert temporal features, while 
the haptic sense can perceive both spatial and temporal information with high precision!. For this reason and 
because it is applied directly on the body, haptic feedback is believed to be the most intuitive augmented sensory 
feedback to describe limb movements, as it provides intuitive spatiotemporal cues directly on the limb thus reduc- 
ing the level of abstraction required from the user to understand the movement”. Here, a haptic display proved to 
be capable of mapping the tactile/proprioceptive cues, describing bipedal walking, onto the skin of the patients’ 
forearm. Patients learned very quickly to link the haptic feedback to the avatar walking movement. 

Multi-sensory training protocols have been reported to be more effective for learning’*. Moreover, the benefits 
of a multi-sensory training phase are maintained further than when a single modality is used”. Similarly our 
protocol for integrating the tactile/proprioceptive feedback with an immersive virtual environment likely helped 
patients to internalize the relationship between a visually observed leg movement and the corresponding tactile 
stimulation in order to recall it once the visualization was removed. In this context, the emergence of this intuitive 
pseudo proprioception has an important implication for future use of brain controlled exoskeletons and other 
prosthetic devices, as the proprioceptive information is essential for controlling locomotion and other autono- 
mous movements”>”®. 

Walking generates large amounts of parallel multisensory information streams that are used in the fine control 
of the gait cycle. Among these inputs, haptic information is essential as proven by our ability to walk with our eyes 
closed and inability to walk when tactile feedback from the legs is disrupted”’. The effects of haptic perception 
during walking are complex as illustrated by the change in walking patterns resulting from a reduction of the 
tactile sensitivity in the foot sole”. 

The importance of tactile feedback can also be appreciated when one examines the SCI patients’ perception of 
their own body. Sitting in a wheelchair after complete loss of sensory motor functions for several years changes 
the perception of one’s own body”. Experiments showed distortion of perceived body parts in SCI patients (they 
perceived a reduction in hip size”’), as well as a decrease of body ownership of the legs and an increase in symp- 
toms of depersonalization*®. SCI patients were also found to consider their wheelchair as a ‘substitution of them- 
selves or a part of their body*!. These observations are not surprising as numerous experiments have shown that 
body representation in the human brain!>*” or primates*’ modulates to incorporate additional limbs™, prosthetic 
limbs**°, tools*’, or virtual limbs'***“°. The so called ‘rubber hand’ illusion*?’, which has been extensively stud- 
ied, illustrates the plasticity of the brain representation of the body or body schema**“!. 

In the present work, we attempted to answer the following questions: do SCI patients incorporate virtual 
legs? Is tactile feedback necessary for this effect to appear, or is the vision of an avatar resembling a human body 
enough? In response, we found that the incorporation effect was very quick and robust. The assimilation effect, 
recorded through well documented psychophysical measurement (CCE!*!8), was observed after 1 minute of a 
congruent visuo-tactile stimulation. Additionally we observed that once the incorporation effect appeared, it was 
maintained for at least a few minutes (as shown by the maintenance of the effect after the last VO block). 

Overall, our results are in concordance with previous findings on incorporation of upper limbs!>**4 and 
lower limbs’? in healthy subjects or amputee patients***°. We also extended these previous observations to SCI 
patients. Moreover, our findings create a bridge between limb incorporation and sensory remapping since our 
subjects integrated the avatar’s legs while receiving tactile feedback on their forearm. 

Finally, reacquiring haptic feedback from the legs recreates the interaction with the external world, or in the 
case of walking, with the ground surface. Spinal cord injured (SCI) patients participating in this study found spe- 
cific vibrotactile patterns that corresponded to the complex sensation of walking on sand, grass or paved street. 
The experimental paradigm allowed the patients to select among all possible discernable combinations of tactile 
parameters (10°000 combinations). The patients selected parameters which are easily clustered, showing that they 
had a clear idea of the sensation they were seeking. Asa result, six out of eight patients performed at above chance 
identification rates. 

One important question regarding these results is whether the patients associated a specific vibrotactile pat- 
tern to a given surface, or whether the vibrotactile pattern elicited a genuine sensation of walking on a specific 
floor. Several elements point towards the second hypothesis. The patients received no training, nor did they 
receive any feedback regarding their selections during the exploratory phase or the inverse task. Second, there 
was a break of at least 20 min between the exploratory phase and the inverse task; a delay larger than the one 
reported for haptic working memory (around 10 seconds). Finally, strong similarities in the selected parameters 
were found among patients, making it highly unlikely for their choices to be a mere repetition of a casual initial 
association. 

Another interesting discovery of our experiment was the parameter that patients relied on the most to identify 
the ground surfaces. When the parameter corresponding to the structure of stimulation was somehow expected 
to be important (patients expected three separated stimulations when walking on a granular surface such as sand, 
a single impact when walking on a paved street and smooth stimulation for grass), it was surprising to see the 
second and only other important parameter was the sensation at the end of the stance. Patients considered the 
sensation of the foot pushing on the floor before starting a swing as a signature for describing the different ground 
types: strong feedback for PST, softer for GRT, lightest for SAT. 

Our experiments highlight the importance of tactile feedback over vision for perception of floor texture. 
Presence or absence of the virtual floor in the virtual simulation did not influence patients’ choices of what they 
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perceived as a certain ground type, although they all judged that the 3D simulation of the virtual grounds were 
realistic. 

The subjective and conscious feeling, or the ‘kind of‘ sensation often referred to as qualia*’, is an important 
aspect of our proposed approach to restore tactile perception. A more direct implementation of floor texture 
display could have been to sample material features from real-world surfaces and remap them through the tactile 
shirt**“°. We have opted for a different approach where naive subjects search for their preferred sensation. 

Lately, the nature of the subject's perception during the use of sensory substitution devices has been ques- 
tioned*®: if someone uses an apparatus to perceive, through vibrators, a stream of visual data, can one conclude 
that the user is ‘seeing with the skin’? While our work does not pretend to permanently solve this debate, we 
believe that our experimental paradigm avoided some of the issues that make interpretation of sensory substi- 
tution devices tedious. Our patients did not learn to discriminate among different stimuli; they reported the one 
they considered closest to their perception of walking on different surfaces. As a result, the majority of patients 
described the sensations as realistic and similar to the one of walking on the corresponding floor type. The 
patients that did not find the sensation realistic were also the ones that reported not remembering walking on 
some of the floor types, suggesting that the sensation found during the experiment was linked to their personal 
experience of walking on these floors prior to the SCI. One patient once spontaneously reported ‘I was walking in 
a happy mood, because I was walking on the beach. 

Altogether our experiments show important emergent positive effects when chronic SCI patients take advan- 
tage of our tactile shirt and highlight the importance of this feedback for BMIs and neuroprosthetics to be clin- 
ically relevant. In addition to the assistive benefits during locomotion, allowing patients to rely less on visual 
feedback, by relying on tactile feedback, patients can look forward instead of looking down at their feet to know 
where they are in their walking cycle. Continuous use of the haptic interface seems to change the patients’ body 
schema by altering their cortical representation of lower limbs through the process of incorporation of the virtual 
avatar legs. 

Interaction with our haptic interface also induced in the SCI patients a more vivid sensation of walking and 
the return of interactions with part of their peri-personal world that had been lost many years prior, i.e. the space 
immediately under their feet. To some degree, by reacquiring the ability to experience contact with the ground, 
and to perceive different types of ground surfaces, our haptic interface provided the patients with a much richer 
lower limb “phantom” sensation. Such an enriched illusion likely contributed to making these patients much 
more amenable to the idea of walking with the assistance of a custom-designed robotic exoskeleton’, since their 
walking experience with this orthosis generated a much more realistic sense of bipedal locomotion that they had 
experienced since their SCI. 


Methods 

This study’s protocol was approved by the ethics committee of AACD (Associacao de Assisténcia 4 Crianga 
Deficiente, Sao Paulo, Brazil) and carried out in accordance with its guidelines. All participants provided written 
informed consent before enrolling in the study. 

All our patients were initially evaluated using the American Spinal Injury Association (ASIA) Impairment 
Scale (modified from the Frankel classification) to quantify the severity level of their spinal cord injury. This scale 
grades SCI from ASIA A, for a complete lesion with no sensory or motor function below the neurological level 
of the injury, to E, for normal sensory and motor functioning'. Seven ASIA A patients and one ASIA B patient, 
all in the chronic phase of the SCI (at least one year after the SCI), were selected as subjects for all experiments 
reported here (see Table S1 for patients’ demography). All our patients had lesions below or equal to the thoracic 
dermatome T4. Accordingly, they all exhibited normal sensory motor functioning in the upper limbs. 

Three different psychophysical experiments were performed in the course of 6 months: a 
pseudo-proprioception test, a cross congruent task and a task involving the simulation of floor textures. Globally, 
these experiments were designed to: (a) provide the patients with an immersive visuo-tactile experience of walk- 
ing; and (b) assess the impact of an augmented somatosensory feedback on the patients’ perception of their own 
body. 

During all experiments patients were seated in their wheelchair while wearing a tactile shirt (see Tactile Shirt 
description). In all the experiments, patients also wore a head mounted display on which a 3D human avatar was 
projected. The avatar could stand and walk and, as it performed these movements, tactile feedback, reproducing 
the touch of the avatar feet on the ground, was delivered on the skin of the patients’ forearm through the employ- 
ment of a haptic display (e.g. the tactile shirt: see Integration of the virtual body avatars with the tactile shirt). 

In the pseudo-proprioception task, we delivered four distinct tactile feedback paradigms emulating different 
features of the avatar’s walk and determined which paradigm was the most intuitive for the patients. 

The Cross Congruent task (CCT)!>*” was used to explore the brain representation boundaries of the body 
schema in our patients after they used the tactile shirt. Finally, by simulating floor textures, we investigated 
whether spatiotemporal changes in the patterns of forearm tactile stimulation could give patients the perception 
of walking on different types of surfaces, like grass, sand or a paved street. Because there is no a priori answer for 
how to stimulate someone's forearm in order to render the complex sensation of walking on different floor sur- 
faces, we proposed a novel approach to search the best vibro-tactile parameters to render the floor types without 
making any assumption on the user's perception. 


Virtual Reality Environment and Setup. Three virtual human avatars (one woman and two men) were 
modified based on free online stock models from Maximo (Maximo Inc. 2015). The virtual avatars were animated 
to walk and stop; animation blending was accomplished using MotionBuilder (Autodesk Inc. 2015). Custom 
written C++ code controlled the triggering of avatar movements, the type of surface where the avatar walked and 
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calculated the interaction of the avatar with the surface. In addition, we integrated the Oculus rift (Oculus VR) 
head mounted display with MotionBuilder using a technique called OpenGl intercept*’. 


Tactile Shirt as a Haptic Display. To deliver artificial tactile and proprioceptive feedback signals, origi- 
nating from the movements of the human 3D avatar, we created a haptic display embedded in the long sleeves 
ofa shirt. This haptic display was named “tactile shirt” and employed eccentric mass (ERM) vibrators to deliver 
somatosensory feedback to the skin of the patient’s forearm. The ERM vibrator consisted of a DC motor rotating 
an eccentric mass at different angular velocities, allowing the generations of various amplitudes and frequencies 
of vibration. ERM frequency and amplitude were coupled, and the maximum stimulation amplitude was reached 
at about 150-250 Hz, which corresponds to the peak response frequency of Pacinian Corpuscles (a type of rap- 
idly adapting mechanoreceptor which is sensitive to mechanical transients and rapid vibrations in the range of 
(~40-400 Hz)) in the human hairy skin*’. Our tactile shirt used three coin-shaped, 2cm diameter, ERM vibrators 
to deliver sensory feedback, reproducing ipsilateral lower limb tactile or proprioceptive signals, to each of the 
patients’ forearms (Fig. la) 

The three vibrators were placed 6 cm apart from each other along patients’ forearms, following the longitudi- 
nal axis of the ulna bone (Fig. 1a). 

Since all our patients had thoracic lesions (T4-T11), in theory none of them should exhibit sensory defi- 
cits in the forearms. Preliminary tests showed that 6 out of 8 patients could discriminate vibrator position with 
50 ms-long vibrations on their forearms (Fig. S3). We decided to move the shirt to the ventral part of the forearm 
for Patient P6, who had difficulties feeling stimuli on the ulna. Patient P1 was found to have difficulties discrimi- 
nating vibration pulses under 70 ms, so longer pulse trains were employed with this patient. 


Integration of the Virtual Body Avatars with the Tactile Shirt. During the experiments, patients sat 
in a wheelchair, wearing an Oculus Rift head-mounted display (HMD) (Oculus, VR) in which virtual legs were 
projected, mimicking the position and orientation of the patients’ own bodies (Fig. 1a). Patients also wore head- 
phones playing white noise to avoid biases due to noise from the vibrators. Prior to the first experiment, we ran 
two tests with the head-mounted display to evaluate if all patients could perceive correctly the 3D Virtual Reality 
(VR). We also evaluated whether any of them experienced any sign of motion sickness (Fig. $5). One subject 
experienced strong motion sickness during the second test while quickly moving the head. Thus, to have a single 
setup suited for all patients, we kept a fixed camera point of view for all patients during all the experiments. 


Pseudo Proprioception Test. We tested two different feedback modalities related to different stages of the 
avatar body locomotion and assessed which one was more intuitive for patients to perceive the position of the 
avatar leg when relying on tactile feedback only: (a) feedback given during the stance phase of the virtual avatar 
legs; (b) feedback given during the swing phase; as well as two directions of tactile stimulation on the forearm: 
proximal to distal (PtD) or distal to proximal (DtP). 

The four combinations given by the two modalities x two directions were tested in separate experiments (ran- 
domized order), with each one divided in four blocks (Fig. S2a). 

The first block lasted 30 seconds during which the patients were looking at the virtual avatar legs through the 
HMD. At this stage, the avatar walked at medium speed of 66 steps per minute; patients did not receive any tactile 
feedback on their forearms. They were asked to look at the avatar legs and imagine that they were their own. 

The second block lasted 1 minute and included the delivery of tactile stimulation on the forearm skin surface, 
in synchrony with the avatar walk and according to one of the four tactile conditions. For the third block, the 
HMD was turned off while the avatar continued walking at the same speed and tactile feedback was delivered 
accordingly. Patients were asked to rely on the tactile stimulation while imagining their own legs moving. This 
block lasted 30 seconds. After the third block, the avatar stopped walking, and after a 20 second break the last 
block started. Here, subjects were instructed to match the upcoming avatar walk, provided through tactile feed- 
back, with a corresponding movement of their arms. The avatar resumed walking, first at 66 steps per minute. 
The walking speed was randomly reduced to 50 steps per minute or increased to 100 steps per minute without the 
user’s knowledge and without specific time patterns (Fig. $2a). A camera filmed the arms of the patient during 
this exercise and compared these to the joint positions of the avatar legs. 

A score, ranging from 0 to 3, was given to evaluate a patient’s performance for each speed. A 0 score was given 
if the patient’s arm movements were not synchronized with the avatar walk at any phase for a certain speed. 
A score of 1 was given if a patient managed to follow the walk at constant speed, but did not manage to follow 
the walk when the avatar’s walking speed changed (for example from medium to high speed). A score of 2 or 3 
was given if the patient managed to follow the avatar walk synchronously during constant and changing speeds. 
Scores differed when patients were synchronous with the avatar walk but exhibited a contralateral inversion (for 
example showing the stance with the left arm during right leg stance and vice versa, score 2), and when patients 
were synchronous and used the correct arm to show the correct leg (score 3). 

We ranked the four tactile paradigms Pi based on the following scoring: 

Score(Pi) = >° patients who had their best score with Pi - © patients who had their worst score with Pi. At 
the end of each test, patients answered a questionnaire (Fig. S2b) and restarted a new test with a different tactile 
paradigm. 


Cross Congruent Task (CCT) test. To measure whether the virtual avatar leg was “incorporated” by the 
patients, we ran an adapted version of the cross congruent task (CCT)!*!**5. This task is based on the observation 
that human subjects are slower in detecting a tactile stimulus on the index finger if a visual distractor appears 
close to the thumb (and vice versa)**. This effect, named crossmodal interference, is stronger if the distractor is 
placed on the same hand as the tactile stimulation than when the visual distractor and tactile stimulation are 
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contralateral. Similarly after active use of a tool, the multimodal interaction between visual stimuli on the tip of 
the tool and tactile feedback on the user’s hand changes™: visual distractors on the tip of the tool led to an increase 
in response time (RT) when the vibration and the distractor were incongruent compared to the congruent case. 
‘These findings were interpreted as revealing the incorporation of the tool as an extension of the subject’s arm, due 
to the projection of cortical visual receptive fields to the distal edge of the tool. 

Patients wore the tactile shirt while observing the 3D avatar through the head-mounted display (Fig. 2a). A 
trial started with a visual fixation cross placed between the avatar’s feet for 1000 ms. Next, a 50 ms vibration was 
randomly triggered in one of four locations: proximal or distal location of the left or right forearms. Light distrac- 
tors (3cm radius 3D blue spheres) appeared 100 ms before the mechanical vibration in one of the four locations: 
left/right toe/heel of the avatar leg (100 ms offset was found to maximize the crossmodal interaction*). Patients 
were asked to indicate, using two keys on a keyboard, whether a vibration was delivered on the front part (distal) 
or in the back (proximal) of the forearm while ignoring the visual distractor. 

Each session started with 5 minutes of training. The experiment started only if patients had >85% accuracy in 
detecting the correct position of vibration during this training. The experience consisted of 5 minute long CCT 
blocks run immediately after subjects experienced 1 minute of observation of the avatar walking (Visual Only, 
VO), either with simultaneous tactile feedback moving from wrist to elbow of the patient’s forearm or when the 
ipsilateral avatar foot was in contact with the floor (Vision + Tactile feedback, VT) (Fig. 2b). 

For each CCT block we tested all 16 configurations of tactile feedback (four positions) and visual distractor 
(four positions). On each block CCT we repeated each configuration four times. We repeated the same experi- 
ment for all patients. 

Response time (RT) was measured and trials with RT longer than 1800 ms or faster than 200 ms were dis- 
carded as well as trials with RT beyond the range of mean +3 x std per (8.5% of the overall trials of all sessions). 


Floor Texture Simulation Test. We investigated the set of tactile parameters that could be used with our 
tactile display in order to induce the illusion of walking on three different ground surfaces: sand (SAT), grass 
(GRT), and paved street (PST). We then compared the parameters obtained for all patients. 

Patients were seated in their wheelchairs wearing the tactile shirt (see Fig. 3a) and a head mounted display. A 
3D human avatar was shown in first person view. The avatar and the virtual environment were rendered using 
MotionBuilder (Autodesk) software. The avatar’s walking was based on motion capture of a healthy subject walk- 
ing at 45 steps per minute. 

We presented a catalog of textures to our subjects. Patients were asked to choose those that best represented 
the sensation of walking on SAT, GTR, and PST. The catalog was created by varying four parameters describing 
the tactile stimulation: the amplitude of the distal vibrator (DV), the amplitude of the middle vibrator (MD), the 
amplitude of the proximal vibrator (PV) and the stimulation timing (ST) (Fig. 3a,b). Each one of these four fac- 
tors had 10 possible levels. The number of possible combinations was thus 10*= 10°000. Vibrator amplitude 1 rep- 
resented the lowest perceived sensation, and 10 the strongest before sensory saturation (both found empirically). 

The stimulation timing was defined by two factors: Duration of the Stimulus (DoS) and the Inter-Stimuli 
Onset Interval (ISOI). ISOI represents the time between onset of one vibrator and the next. The DoS was chosen 
to satisfy the following relation: DoS = Stance Duration - 2xISOI; where Stance Duration was fixed to 2000 ms 
throughout the entire session. ISOI varied between 100 ms and 820 ms. An ST level 1 referred to the shortest ISOI 
and longest DoS (Fig. 3d) and corresponded to three long vibrations delivered almost simultaneously. For an ST 
of level 10, the onset asynchrony was longer than the stimulation duration, resulting in three short and distinct 
vibrations (not overlapping). Between these two extremes some levels of ST produced a continuous moving touch 
illusion known in haptics as apparent movement"®. 

Each experiment started with patients seated in front of a table wearing the head-mounted display to observe 
the 3D avatar and with their arms placed on a table (Fig. 3a). A thick tape was used to delineate two square areas 
of 50 x 50cm? on the table. Patients were asked to keep the left hand inside the left square and right hand in the 
right square. A tracking system recorded the position of both hands in the two referential areas defined by the 
two squares. As patients moved their hands over the two square areas, they received a particular pattern of tactile 
stimulation, defined by the four factors described above (Fig. 3b). More specifically, planar, Cartesian coordinates 
of the left and right hand were mapped onto four tactile parameters. Hence, each spatial position inside the two 
squares was assigned with a particular pattern of tactile stimulation to the patient’s forearms. Axes were rand- 
omized at each trial. 

The session started with a 15-minute training phase. The experimenter asked the patient to find the tactile 
feedback that represented best for her/him walking on sand or grass or paved street. Patients freely explored the 
2D spaces defined by the two squares. As they explored this virtual “tactile space’, patients were asked to observe 
the avatar walking on a black floor. Tactile feedback was delivered via the tactile display during the stance phases 
of the avatar on the ipsilateral arm. 

Next, an exploratory phase started where the same procedure was repeated 40 times per surface (a total of 120 
trials), in a randomized order. To avoid patients learning the position of a certain texture on the table, the four axes 
on the table were randomized for every trial (four possible axis configurations for left side, four for right side and 
all possible permutations of the four parameters = 4 x 4 x 4! =384 possible configurations, Fig. 3c). Here, instead 
of the experimenter announcing the floor type, a cube appeared for 5 seconds in the Virtual environment (VE) at 
the beginning of each trial: red for SAT, green for GRT and blue for PST. For half of the trials (randomized order), 
immediately after the cube disappeared from view, the corresponding floor - sand, grass, or paved street - was 
displayed in the VE (Fig. 3a). For the other half of trials, the floor stayed black. Patients had 2 minutes to explore 
the table (and thus the catalog of textures) and confirm that the correct sensation was identified by raising their 
right hand by 10cm. The task was the same in the visual presence or absence of the corresponding floor in the VE. 
After confirmation, the avatar was stopped. Following a 2-5 second inter-trial time, a new colored cube appeared 
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and a new trial started. The trials were divided as follows: first block of 36 trials, a break of 5 minutes; a second 
block of 36 trials, a break of 45 minutes; and a third block of 48 trials. Inside a block there were always the same 
number of trials with and without floor and the same distribution of surface types. 

After the exploring phase was concluded and patients had a 20 minute break, the next step of the experiment 
started. The experimenter played back in the haptic display applied to the patients’ forearms the 120 textures 
chosen during the searching phase. Patients had to say which floor type the avatar was walking on or say ‘I don't 
know. The patient held their arm on the table and, using the head-mounted display, observed the avatar walking 
onan empty black floor. This phase was named the inverse task. At the end of the first session a questionnaire was 
administered: 

Q1) During the experiment I had the impression of walking on SAT, GRT, PST. 

Q2) Iremember the sensation of my feet on SAT,GRT, PST. 

Two months later we ran a control experiment with all patients. The patient observed the avatar walking on 
the three different floors for 15 minutes. No tactile feedback was used. The session was followed by a questionnaire 
containing question Q1 and the following question: 

Q3) The floor of type SAT/GRT/PST I saw in the head-mounted display was visually realistic. 
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Abstract 


Spinal cord injury (SCI) induces severe deficiencies in sensory-motor and autonomic func- 
tions and has a significant negative impact on patients’ quality of life. There is currently no 
systematic rehabilitation technique assuring recovery of the neurological impairments 
caused by a complete SCI. Here, we report significant clinical improvement in a group of 
seven chronic SCI patients (six AIS A, one AIS B) following a 28-month, multi-step protocol 
that combined training with non-invasive brain-machine interfaces, visuo-tactile feedback 
and assisted locomotion. All patients recovered significant levels of nociceptive sensation 
below their original SCI (up to 16 dermatomes, average 11 dermatomes), voluntary motor 
functions (lower-limbs muscle contractions plus multi-joint movements) and partial sensory 
function for several modalities (proprioception, tactile, pressure, vibration). Patients also 
recovered partial intestinal, urinary and sexual functions. By the end of the protocol, all 
patients had their AIS classification upgraded (six from AIS A to C, one from B to C). These 
improvements translated into significant changes in the patients’ quality of life as measured 
by standardized psychological instruments. Reexamination of one patient that discontinued 
the protocol after 12 months of training showed that the 16-month break resulted in neuro- 
logical stagnation and no reclassification. We suggest that our neurorehabilitation protocol, 
based uniquely on non-invasive technology (therefore necessitating no surgical operation), 
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can become a promising therapy for patients diagnosed with severe paraplegia (AIS A, B), 
even at the chronic phase of their lesion. 


Introduction 


Spinal Cord Injuries (SCI) cause a wide array of disabilities with devastating motor, sensory, 
and autonomic deficits that impair the functional capacity of patients to perform routine living 
and working activities. SCI also leads to significant impairments in the patient’s quality of life 
(QoL) [1], his/her body image [2] and sexuality [3]. As such, SCI rehabilitation must consider 
the patient’s physical, emotional, social and affective life aspects [4], aiming at promoting the 
patient’s physical independence and autonomy, while promoting the reintegration of the indi- 
vidual into society. Epidemiological studies on SCI [5] reveal a global incidence, considering 
traumatic or non-traumatic etiology, around 250,000 to 500,000 new cases added every year. 

The severity of clinical impairment and recovery prognosis depends on the SCI level [6], 
the extension and mechanism of injury, presence/absence of residual spinal cord fibers and 
pre-existing clinical comorbidities. In the US, the most common rank observed one year after 
the original SCI is AIS A (34%), which includes patients with complete neurological loss SCI 
(but not necessary anatomically complete lesions [7—-9]) [10]. The majority of such AIS A cases 
involves lesions at the thoracic level (67%) [10]. Studies show a considerable amount of sponta- 
neous improvement during the first year following the lesion [11,12], and stagnation at the 
chronic phase. For example, following 26 weeks after the original lesion, the rates of motor 
score recovery for AIS A patients drops to less than 1 point (on a scale where complete paraple- 
gia is 0, and normal function is 50). Between 1 and 5 years after the SCI injury, only about 
3.5% of AIS A patients are upgraded to AIS B, 1.05% to C and 1.05% to D [13,14]. These statis- 
tics indicate that the most commonly observed cases of chronic paraplegic SCI are represented 
by AIS A patients who have very little chance of spontaneous neurological recovery at the 
chronic phase of the injury. 

The rehabilitation process with SCI patients involves both learning of tasks as in the use of 
a wheelchair, and a variety of compensation mechanisms to recover from lost motor functions 
[15]. The use of stem cell therapy for complete SCI has also been investigated in recent years, 
both experimentally and clinically, but despite some interesting outcomes, it has not yet been 
established as a standard effective treatment [16]. Potential new treatments for controlling 
neuropathic pain, spasticity, bladder, and intestinal functions have also been extensively stud- 
ied. Those include electrical spinal cord stimulation, chemical neuromodulation [17], drug 
delivery pumps with catheters inserted in the subarachnoid space [18] and sacral stimulators 
[19-21]. 

Despite important recent improvements in rehabilitation techniques, the chances of neuro- 
logical recovery for motor complete SCIs (AIS A/B) remain low. Indeed, whereas a number of 
neurorehabilitation protocols have induced some level of neurological recovery in motor 
incomplete SCI patients (AIS C/D) (including stepping training [22,23], operant conditioning 
[24,25] or functional electrical stimulation [26]), improvement in motor complete SCI has 
been principally observed through compensatory mechanisms [15]. Only recently, a few stud- 
ies in rats [27] and humans [28] have shown partial motor recovery (neurological and func- 
tional) in severe cases of SCI, following training with invasive epidural stimulation (see [29] 
for a review) or invasive pelvic nerve stimulation [30]. Notably, clinical improvement was 
noticed when such invasive stimulation was paired with direct patient control of the stimulat- 
ing system, via a brain-machine interface [31] (see [32] for a review). 
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In a previous study, we have reported significant levels of neurological recovery in eight 
chronic motor complete (3-13 years post-lesion) SCI patients, after 12 months of training 
with a multi-step, non-invasive neurorehabilitation protocol [33]. Named the Walk Again 
Neuro-Rehabilitation (WA-NR), this protocol combines locomotion training, brain-machine 
interfaces (BMIs [34]) and visuo-tactile feedback. In the WA-NR protocol, SCI patients learn 
to use their brain activity, recorded via EEG, to control the locomotion of virtual human ava- 
tars and robotic gait devices. To close the control loop, continuous streams of tactile feedback 
are delivered to the skin of patients’ forearms, via a haptic display referred to as the tactile shirt 
[35], in synchrony with regular visual feedback. 

In the present study, we report a detailed analysis of a subgroup of patients from our origi- 
nal study, including six chronic (3-13 years post-lesion) AIS A patients and one AIS B (6 years 
post-lesion) SCI patient, who continued to train under the WA-NR protocol for a period of 28 
months. As a result of this training, we observed significant levels of neurological recovery 
which included improvements in nociceptive, tactile, and proprioceptive function, a marked 
improvement in multiple visceral functions (bladder control, bowel function and sexual func- 
tions in some patients), and significant gains in voluntary motor control of the lower-limbs 
(confirmed by both clinical evaluation [6], and neurophysiological measurement (EMGs)). 
The patients’ anatomical lesion level was revealed by MRI analysis of their SCI. Overall, we 
observed that the observed sensory and motor improvements occurred in the areas of the 
body innervated by portions of the spinal cord below the original anatomical SC lesion. As a 
consequence of such major sensory recovery, we identified a complex reorganization in the 
patients’ perception of their bodies. Finally, we observed that this partial neurological recovery 
induced significant improvement in both the psychological and physical aspects of the 
patients’ self-report on their quality of life [36]. 

By the end of the training, all seven patients had improved their AIS grade, representing, as 
far as we can tell, the largest cohort of chronic complete paraplegic patients reported in the lit- 
erature to exhibit a consistent partial neurological recovery, following a purely non-invasive 
neurorehabilitation approach. 


Materials and methods 
Participants 


Eight paraplegic patients (Table 1), 27-38 years old, with traumatic and chronic SCI (lesion 
3-13 years before onset of the training) at thoracic level (T4-T11), participated in the current 
study: seven patients, from hereon designated as Group 1 (GR1), including six AIS A patients 
(P1, P3, P4, P5, P6, P8) and one AIS B (P2), followed the WA-NR protocol [37] for a total of 
28 months of training. One subject (P7, AIS A) discontinued the training after 12 months and 
is therefore discussed separately. The participants in the current study are the same that partic- 
ipated in a previous protocol reported by our group [37]. For clarity, the convention used for 
subject names is the same in both studies. 

Initially, the level and the grade of each SCI was estimated, using the ASIA standard assess- 
ment [6] (Table 1). This measurement was done within the first and the third year post-lesion 
and is referred to in this paper as the baseline. Patients’ original AIS classification and neuro- 
logical status were confirmed by our medical team at the onset of training (referred to as time 
0). Details of patients ASIA score is reported in S1 Fig. To complete our neurological investiga- 
tion, we examined the patients’ spinal cord using MRI. This measurement was performed 
once at the end of the protocol. Patient P1 was excluded from the MRI examination for safety 
reasons (presence of undefined material used for arthrodesis). Data from patient P7 were not 
recorded due to protocol discontinuation. For three out of the six tested patients, namely P2, 
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Table 1. Patients’ demography. 


Subject | Training period | Sex | Age | AIS’ | Lesion Level | Lesion Level MRI° | Time Since lesion (years) | Etiol* | Time since baseline ASIA 
(months) Clinic” months” 
Right Left 
Pl 28 F 32 A T10 T1l No data 13 Cc 142 
P2 28 M 26 B T4 T4 T1-T3 6 Cc 54 
P3 28 M 32 A Tll T10 T10-L1 5 oO 30 
Pp4 28 M 38 A T8 T8 T7-T9 5 Cc 40 
P5S 28 M 36 A T7 17 T7-T10 3 Cc 22 
P6 28 M 29 A T4 T4 T3-T5 8 C 54 
P7 12 M 27 A T5 T7 No data 6 C 62 
P8 28 F 29 A T1l T1l T4-L4 9 C 110 


1) ASIA neurological standards evaluation 
2) Clinical lesion level (AIS) 


3) Anatomic lesion level (MRI guided). 


4) Traumatic Etiology. C: Closed Trauma; O: Opened Injury. 


5) The first measurement is done by the clinical institution that followed them before their enrollment in our protocol done at n months before the onset of our training. 


https://doi.org/10.1371/journal.pone.0206464 .t001 


P3, and P4, MRI images revealed the existence of some degree of spinal cord continuity at the 
lesion level (S2 Fig and S1 Video). MRI analysis was partially compromised by artifacts due to 
the presence of metallic implants in two cases (P5, P6) and by the complexity of the injury in 
one case (P8), making it difficult to distinguish between neurological and pathological tissue. 

To assess the effect of our intervention, we clinically evaluated GR1 patients eight times: at 
the protocol onset, and after 4, 7, 10, 12, 16, 22 and 28 months of training. Patient P7 was eval- 
uated during the first year and then once at the end of 28 months, after 16 months of training 
discontinuation. 


Inclusion criteria 


Subjects were adult paraplegics, grade AIS A (complete), B (motor complete) (3) with trau- 
matic SCI at the thoracic level, at least 6 months before the onset of the study, with the absence 
or offset comorbidities, and emotionally stable. We excluded patients with non-traumatic SCI, 
decompensated comorbidities, a degree of spasticity exceeding a score of 2 (on the Ashworth 
scale), degree of osteoporosis (T- score) < -4, and presence of joint deformities, fractures, pres- 
sure ulcers grade > 3, peripheral neuropathy of the upper limbs, brain injury, degenerative 
neuromuscular injury amputation of upper or lower limbs, pacemakers (cardiac/neural), 
cephalic (cranial/brain) implants and with emotional instability. 


Design of experiment 


The WA-NR protocol [33] consisted of two main classes of training: active locomotion train- 
ing (TR-LOC) and BMI-based neurorehabilitation exercises (TR-BMI). Progressively more 
complex stages, as part of a multistep strategy, were employed during application of the 
WA-NR protocol [33], assuring that patients had enough time to acquire upper limb and 
trunk strengthening, cardiovascular stability, as well as emotional adaption to the experience 
of orthostatic posture. The TR-LOC included training with both a robotic gait training device 
(Lokomat) and a body weight support system (ZeroG). 

The steps for the TR-BMI training included BMI control of a virtual avatar while the patient 
was in a seated position, BMI control of an avatar while patient was in an orthostatic position, 
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Table 2. Average monthly training. 


Periods 
(Months range) 


i 
iS 
- 
N 
q 
_ 
o 


10-12 12-16 16-22 22-28 


TR-BMI 


Pl 


P2 


125} 


P4 


P5 


P6 


P7 


TR-LOC 


P8 
Pl 


P2 


P3 


P4 


P5 


P6 


P7 


P8 


PWT PLWH [YH | WlwW LW PHI nD i) nin ion) & 
SIO l/Ol/Ol/Ol]/O]/O Ow Jo] |, P [o ] on] [ow 
NIN [wl fwlwlre iw fe fe de Je de lr lr je 
o;jo!;}o;o;o};o;o;o;yo;jo;o;ojojo;o;o 
FLOR IM IN IN IN |N JK LOOK Je IN |Ole 
td Re 
SF 1OTNM RIN] Re tele lolol;ol;oljol/o;o|;o 


Values are average numbers 


of sessions per month for BMI-based neurorehabilitation exercises (TR-BMI) and active locomotion training (TR-LOC). 


https://doi.org/10.1371/journal 


-pone.0206464.t002 


BMI control of the Lokomat, and BMI control of an exoskeleton (all steps of this training are 
detailed in [33]). Table 2 shows the average number of training sessions for each patient. On 
average, during the first four months, patients had two interventions per week and once per 
week for the rest of the protocol. The period 10-12 corresponded to a break period for all the 
patients, with no training involved. 

Active locomotion training. TR-LOC activities included locomotion training with a 
body weight support (BWS) system, robotic gait therapy device on a treadmill (Lokomat, 
Hocoma), and training with an over-ground fixed track BWS system (ZeroG, Aretech LLC). 
During gait training, subjects were guided to attempt to execute lower-limb motor tasks 
actively, despite the SCI. For the Lokomat training, the physiotherapist verbally instructed the 
patients to try to actively perform lower limb movements, in conjunction with the computer- 
generated orthosis movements, which included flexion and extension of hip, knee, and ankle. 
Each session was distributed in blocks, during which specific movements were trained; for 
instance, hip flexion during swing phase. The device provided real-time biofeedback regarding 
the patient’s joints torque. BWS was limited by the maximum knee extension, without joint 
collapse during the stance phase of gait (up to 80% of BWS, avoiding further reductions in this 
device). Guidance force was fixed at 100%, and treadmill speed was set between 1-1.5 km/h, to 
promote the safest possible training environment. During ZeroG training, subjects wore leg 
orthosis (for joint stabilization: hip, knee, ankle), and employed a wheeled triangular walker, 
while being assisted by the physiotherapist. BWS was progressively decreased from 75% to 
30% throughout the sessions (bone densitometry guided the lower BWS level). The training 
promoted postural control, dynamic balance control, cardiovascular conditioning, upper limb 
and trunk strengthening, lower limb voluntary activation during stance and swing gait sub- 
phases. 

BMI-based neurorehabilitation exercises. A 16-channel EEG cap was used for all the 
TR-BMI sessions. Overall, we implemented and tested two BMI control strategies. The hybrid 
state machine (Fig 1A) strategy employed 16 channel EEG recordings over the arm motor 
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cortex area (Fig 1B) and two channel EMG recordings of the subject’s upper arms. Subjects 
used left and right arm motor imagery to select specific actions in a state machine (Fig 1C) and 
two stage EMG activation to confirm the selection and the state transition to trigger the action. 
This two-step confirmation strategy ensured that the movement of the avatar or the robotic 
legs was not triggered by mistake in case a false positive was detected at the EEG decoding 
phase. Patients could use this strategy to control sit/stand-up and walk/stop states. Alterna- 
tively, the same approach was used for patients to trigger walk/stop and kick states. 

The single leg control (SLC) protocol employed 16 channel EEG recordings, clustered 
around the medial longitudinal fissure, meaning that it was more densely focused over the 
putative leg motor representation area of the primary sensorimotor cortex (Fig 1B). Subjects 
used left or right leg motor imagery to trigger the stepping of the corresponding limb. 

Both BMI strategies were tested to control a 3D virtual avatar and a robotic leg actuator 
(Lokomat or a custom-built exoskeleton [38]). An array of vibrotactile actuators placed on the 
patients’ forearms provided online artificial proprioceptor/tactile feedback regarding the posi- 
tion of the legs (virtual or robotic) during the locomotion and the contact of the robotic actua- 
tor feet with the floor. This feedback was delivered to the patients’ forearms using a portable 
haptic display developed by our team and called the “tactile shirt” [35]. An LED display, inte- 
grated into the Lokomat and the exoskeleton, informed the patient about the status of the 
experiment and the outcome of the EEG classifier. 

For both BMI strategies, we used linear discriminant analysis (LDA), using features 
extracted by a six-dimensional common spatial pattern (CSP) to construct an EEG classifier. 
The first BMI strategy was tested during the first 6 months of training and the second one for 
the rest of the protocol. 


Statistics 


Given the small samples of subjects, when possible, the details per subject are shown rather 
than a group average. Graphs with group scores report group mean and SEM. 

Correlation analysis. To study the main factors that influenced the observed neurological 
improvement in our patients, we performed a correlation analysis between the improvement 
rate and training related factors (TR) and external factors (EX). The improvement rate was cal- 
culated as the difference of the AIS score (motor or sensory) between two consecutive AIS 
assessments. We considered the eight measurements done by our team for patients in GR1 (at 
the onset of training and after 4,7,10, 12, 16, 22 and 28 months) and the six measurements for 
patient P7. 

The training-related factors were: the number of training hours of locomotion training 
(TR-LOC), and the number of hours with the BMI-based training (TR-BMI) (Table 2). The 
external factors were the SCI height (EX-SCH, listed from cervical to sacral, i-e., first cervical 
C2 = 1 and last sacral dermatome S4-S5 = 28), the time since the lesion (EX-TME), and the 
patients’ age (EX-AGE). For the correlation analysis between the improvement rates and the 
experiment factors, the Pearson correlation coefficient and p-values of the correlation are cal- 
culated with the standard corrcoef Matlab function. 


Clinical assessments 


Magnetic resonance imaging (MRI). Spinal cord MRI scans, using a 1.5T GE-Genesis 
equipment, with gadolinium-based intravenous contrast, were collected 28 months after the 
training onset. Images were obtained in axial, sagittal and coronal planes, and in T1, T2 and 
FIESTA (Fast Imaging Employing Steady-state Acquisition) sequences. Submillimeter cuts 
were evaluated by a radiologist blinded to the experimental paradigm to detect possible 
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Fig 1. Brain-machine interfaces control strategy. (A) We have developed and validated two BMI strategies: the hybrid EEG and EMG 
state machine control (HSM) and the EEG single leg control (SLC). Both strategies were used to control the actuation of a virtual avatar 
and a robotic gait device (Lokomat or exoskeleton). A portable haptic device was used to inform the user of the position of the virtual or 
robotic leg and the contact of these actuator’s feet with the floor in real time [35]. (B) For both BMI strategies, a 16 channel EEG cap was 
used. Electrodes were clustered over the arm area of the sensorimotor cortex for HSM and leg area for the SLC strategy. The ground and 
reference electrodes are reported in gray and light blue respectively. (C) The control strategy for the HSM (left panel) is based on 
navigation of a state machine (middle panel), using motor imagery, and a two-step EMG confirmation, using isometric muscle 
contraction of the biceps (IMC). For example, when the subject is in a standing position (Stop/Safe sate) and wants to start walking, (s)he 
will imagine moving the left arm and confirms the choice with a left bicep IMC. The subject has then to produce a right bicep ICM to 
trigger walking. The SLC strategy uses the decoding of leg motor imagery through EEG signals. If left motor imagery (LMI) is detected 
the left step is triggered. Once in this position, if right motor imagery is detected, the right step is triggered, and if no state is detected for 5 
seconds, the actuator (the avatar or the robotic gait device) returns to the idle position. 


https://doi.org/10.1371/journal.pone.0206464.g001 


residual neural fibers at the level of the SCI. An additional 3D reconstruction of the spinal 
cord was obtained using the following method: (1) a spinal cord toolbox [39] was used to align 
the slices of the spinal cord; (2) centerline image was set manually on each axial slice, and (3) 
segmentation was obtained with sct_propseg function within the cerebrospinal fluid and con- 
firmed by an experienced doctor. The three-dimensional mesh was rendered with ITK-SNAP 
software. 

Neurological evaluation: Sensory and motor scores. A somatosensory score was calcu- 
lated by summing left and right dermatomes with normal (coefficient 2) and altered (coeffi- 
cient 1) sensation for each patient [6]. Then, the total somatosensory improvement was 
calculated as the difference between the score after n months of training and the initial score. 
The motor evaluation followed the standard AIS motor assessment methodology [6]. Motor 
evaluation was conducted through a functional examination of 12 muscles, among them five 
key muscles (rectus femoris proximal portion, rectus femoris distal portion, tibialis anterior, 
extensor hallucis longus, gastrocnemius) and seven non-key muscles (hip adductors, gluteus 
maximus, gluteus medius, medial and lateral hamstring,flexor hallucis longus and extensor 
digitorum longus). Note that the presence of voluntary anal contraction and/or presence of 
any motor function (grade 1 or above) more than three levels below the motor level on a given 
side will determine if a patient is AIS B or AIS C (6). The motor score described the level of 
voluntary strength below the SCI level for each patient, ranging from 0 (absence of contrac- 
tion) to 5 (for normal contraction, produced against gravity and strong opposing force). The 
lower extremity motor score (LEMS) was obtained by summing all key muscles (score for a 
healthy subject is 50, i-e., five key muscles, with a maximal score of 5, bilaterally). 

Proprioception measurements. For the proprioception evaluation, the experimenter per- 
formed individualized joint mobilizations at the lower limbs: hip flexion and extension (F/E), 
knee F/E, ankle dorsiflexion/plantarflexion, hallux F/E and toes (2”* to fifth) F/E. The patient 
was blindfolded and in a horizontal supine position, while the examiner performed manual 
joint mobilization at an approximately angular speed of 60°/second. For the hip flexion, we 
lifted the leg up maintaining the knee extended and the lower limb aligned, to exclusively 
move the hip towards flexion, up to 60° (initial position = 0°). For the hip extension, the initial 
position is 60° of flexion, lowering the leg towards the clinical table. To perform knee mobili- 
zations, we kept the femur lifted with a hip flexion of 30°. For knee flexion, we moved the tibia 
from extension position 0° towards the clinical table, performing 30° of knee flexion. For knee 
extension, we moved the tibia from 30° of flexion, back to complete extension 0°. For ankle 
dorsiflexion, we moved the foot up from neutral position (0°) to 10° of dorsiflexion. For ankle 
plantarflexion, we moved the foot down from the neutral position (0°) to 40° of plantarflexion. 
For hallux flexion, we moved it down from the neutral position to 45° of flexion. For hallux 
extension, we moved it up from the neutral position to 45° of extension. For toes flexion, we 
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moved them down from the neutral position to 30° of flexion and finally for toes extension; we 
moved it up from the neutral position to 40° of extension. 

Patients were instructed to describe when (s)he could perceive the stimulus. We performed 
10 different mobilizations for each leg. Then, we calculated the proprioception score as the 
number of joints in which mobilization was perceived, meaning that a maximum score of 20 
corresponded to a subject with proprioceptive function present at the entire lower limb area. 

Vibration measurement. For this assessment, a vibrating diapason was placed on differ- 
ent bony prominences, including: the ribs; anterior superior iliac spine (hip); patella (knee); 
medial and lateral malleolus (ankle); hallux, calcaneus bone and the sole of the foot (foot), 
while patients remained blindfolded in a supine position. Patients described the vibration sen- 
sations they experienced after the stimuli were delivered to the trunk and later to the lower 
limbs, following a random sequence that included bilateral areas like the anterior superior iliac 
spine, patella, medial and lateral malleolus, calcaneus, hallux, and the sole. Patients were asked 
whether they could feel the vibration and to describe the stimulus, considering two parameters: 
location (which area) and side of the body (right or left). The vibration score for each patient 
was obtained by summing the regions (rib, hip, knee, ankle, and foot) where the patient con- 
firmed perceiving the stimulation (independently from their ability to place it on the correct 
body part or not). Note that the vibration and the confusion map should be considered as a 
whole to describe each patient’s vibration perception, the first one indicating if the subject can 
detect the presence of the stimulus, and the second describing the ability to localize it properly. 
To avoid a mechanism of learning by association, no feedback on the outcome was given to 
the patient; the experimenter did not tell the patient if his/her answers were correct or not. 
Also, throughout the assessment, the patients stayed blindfolded and in a supine position. 

Autonomic Visceral function evaluation. Visceral recovery was measured using ques- 
tionnaires that were based on ISCOS datasets for urinary, intestinal and sexual functions [40- 
43], in addition to direct clinical measurements included in the AIS sacral exam to evaluate 
anal sphincter function [6]. 

EMG recording and analysis. EMG activity was recorded using bipolar surface elec- 
trodes, amplified with actiCHamp amplifier (Brain Vision LLC, Morrisville, NC), and digitized 
at 2000HZ. Openvibe, an open-source software, was used for data collection [44]. A linear 
EMG envelope was obtained by rectifying and low-pass filtering the EMG signal, using the 2" 
order anticausal Butterworth with 1Hz cutoff frequency. During the test, patients were sus- 
pended by the body weight support (Hocoma AG, Switzerland) while their legs were attached 
to the Lokomat working in passive mode (motors off and backdrivable). The physiotherapist 
verbally instructed patients to flex and extend the left or right hip for 4 seconds and to relax in 
between tasks. 

Quality of life assessment WHOQOL-BREF. The WHOQOL-BREF [36] is a cross-cul- 
turally valid self-assessment questionnaire with four Quality of life (QoL) domains: physical, 
psychological, social relationships and environment (see S1 Table). The score ranges from 1 to 
5, on a Likert scale, according to the graduation of agreement or disagreement of the partici- 
pant. The scores pointing 0 indicate poor QoL and 100 indicate good QoL. Note that higher 
values in WHOQOL-BREF always mean a positive effect; as such, a high score for the question 
‘How often do you have negative feelings such as blue mood, despair, anxiety, depression?’ 
means low occurrences of negative feelings in the patient. 


Study approval 


Our protocol was approved by both the local ethics committee (Associa¢ao de Assisténcia a 
Crianga Deficiente, Sao Paulo, Sao Paulo, Brazil #364.027) and the Brazilian Federal 
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Government Ethics Committee (CONEP, CAAE: 13165913.1.0000.0085). All participants 
signed written informed consent before enrolling in the study. 


Results 
Sensory improvement 


Fig 2A depicts the normal tactile (dark pink) and altered (light pink) sensitivity (hyper or 
hypo-sensitivity)-which defines the zone of partial preservation (ZPP)-exhibited by each 
patient at the onset of the training. In this figure, each patient’s gains regarding normal (light 
pink) and altered (light blue) somatic sensitivity are also plotted for 12, 22 and 28 months of 
training. Overall, five patients (P1, P3, P4, P6, P8), all diagnosed with complete paraplegia (AIS 
A) at the onset of the training, recovered nociception in their trunk and lower limbs. The most 
dramatic improvement was observed in patient P6, whose T4 SCI occurred 8 years prior to 
our study. At the onset of training, this patient’s ZPP extended to T5-T6. However, 22 months 
after training onset, P6’s ZPP increased by 15 dermatomes on the right side, and 16 derma- 
tomes on the left side. Patient P2 (T4, AIS B, motor complete) recovered normal nociceptive 
sensation in three to six dermatomes below the original SCI. Patient P5 regained altered sensa- 
tions in the sacral dermatomes (S4-S5) 28 months after the training onset, even though he did 
not exhibit sensitivity below T7 on both right and left sides at the onset of training. 

The average increase of ZPP for the GR1 patients (excluding P2 and therefore only AIS A 
patients) is shown in Fig 2B. On average, patients recovered partial nociceptive sensation in 
areas situated 11 dermatomes below their original lesion (11.17 on the right side and 11.33 on 
the left side). Overall, the patients’s sensory score improvement rate for nociception during the 
28-month WA-NR period was significantly higher than the spontaneous recovery rate regis- 
tered before the onset of the training (calculated as the difference between the score registered 
at onset of the training and the baseline score) (Fig 2C, Wilcoxon rank sum test, n = 7, 
p<0.001). 

Fig 2D displays the temporal evolution of the sensory score improvement for GR1 patients 
(black line), compared to the score at the onset of the training for nociception (NC), crude 
touch (CT), pressure (PR) and temperature (TE). The GR1 patients exhibited a consistent and 
significant increase between the 4" and 28" month of training in NC (12.15 + 2.42; mean+ SE, 
Wilcoxon test, p<0.001), CT (10.14+1.16, p<0.001), PR (14.28+2.13, p<0.001), but no signifi- 
cant improvement in TE (0.71+0.44, p>0.1). 

Periods with reduced training (between the 10" and 12" months) also coincided with the 
absence of changes (NC) or reduction (CT, PR) in the group averages. Consistent with this 
observation, patient P7’s scores (continuous red line) stagnated between the 12" and 28" 
months. Note that during the first year, this patient had exhibited an improvement rate above 
the GR1 average in NC, CT and PR sensations. 


Proprioception, vibration sensitivity recovery, and body schema 


All patients exhibited a significant recovery in lower-limb proprioceptive sensitivity (Fig 3A). 
This assessment was first introduced 4 months into the training protocol. The mean proprio- 
ception score at the end of the 28 months of training was 17.14+1.89 (mean + SE) over 20, in 
complete contrast with the measurement at the 4" month of the training, where no patient 
was able to report proprioceptive sensation in any of the tested joints (average group 

score = 0.0). After 28 months of training, six out of seven patients recovered proprioception 
up to the ankle and five patients up to the toes (see S3 Table). Patient P7’s proprioception 
improved up to the ankle after 10 months of training and 18 months later, despite no further 
training, this patient still preserved this improvement (Fig 3A). 


PLOS ONE | hitps://doi.org/10.1371/journal.pone.0206464 November 29, 2018 10/33 


75 


oe 
® . PLOS | ONE Sensory-motor, visceral and psychological improvement in paraplegics 


A. Gained nociceptive sensation per patient 


Normal sensation 
at the onset of training 


Altered sensation 
at the onset 
of the training 


Gained normal 
sensation 


Gained 
altered sensation 


B. Mean dermatome improvement for 
nociceptive sensation for AIS A patients 
(n=6) 


Altered sensation 


ZPP Zone of partial preservation 
+e Normal sensation 
R 
15 
10 
5 


+12 +22 +28 +12 +22 +28 


Nb. dermatomes 


0 4 8 12 16 20 24 28 


Delta nociception score/month 


months 
Sponta- WA-NR 
-0.2 neous (28 months) 
recovery 
D. Sensory score improvement Avg - 
—— a —o— =—s = = fe) 
GRI1 (n=7) er training 
Nociception Crude touch Pressure Temperature 
20 
Se 
3 £ 10 
8 ns 
se ; ae 
a 0 
0 4 8 12 16 20 24 28 0 4 8 12 16 20 24 28 0 4 8 12 16 20 24 28 0 4 8 12 16 20 24 28 


months 


PLOS ONE | https://doi.org/10.1371/journal.pone.0206464 November 29, 2018 11/33 


77 


oe 
(6) ° PLOS | ONE Sensory-motor, visceral and psychological improvement in paraplegics 
“ 


Fig 2. Sensory improvement following the WA-NR training protocol. (A) Patients’ normal (dark pink) and altered nociceptive 
sensations (light pink) areas measured with the standard ASIA assessment [6] at the onset of the training. Dermatomes in dark blue 
represent body areas where patients recovered normal sensations after our training; areas in the light blue are those where patients gained 
altered sensation (reported after 12, 22 and 28 months of training). (B) Mean + standard error of the mean (SEM) for gained dermatomes 
with normal (dark pink), altered (light pink) and zone of partial preservation (ZPP) with nociceptive sensation for GR1 patients except P2. 
(C) Mean+SEM of the nociception improvement rate during the period of the WA-NR (difference of score between the onset and the end 
of the training, normalized by the number of months between the two evaluations) compared to the mean improvement rate before the 
training (difference of score between baseline and onset measurement normalized by the number of months between the two 
measurements). (D) Means + SEM for the gained sensory score for each assessment of GR1 patients for nociceptive sensation, crude touch, 
sensitivity to pressure and temperature compared to the onset of training. Patient P7 stopped the training after 12 months and was 
therefore reported separately from the group average (in red). The score obtained after 4 months of training is compared to the one 
registered after 12 and 28 months of training (t-test, * P<0.05, ** P<0.01, *** P<0.001). 


https://doi.org/10.1371/journal.pone.0206464.g002 


In parallel, the patients’ vibration sensitivity was enhanced markedly between the first mea- 
surement done after 4 months of training (4.0+0.94, mean +SE) and the 28" month of training 
(7.7140.97) (p<0.05, Wilcoxon test) (Fig 2B). Different from the other sensory modalities, we 
observed a notable decrease in patient P7’s vibration score after protocol discontinuation. 

As part of our evaluation protocol, patients were asked to report either the presence or the 
absence of sensation, and to identify the location on the body where they felt the vibration (Fig 
3C and 3D). As training progressed, patients started to recover vibration sensations in more 
distal parts of the body. However, the newly recovered sensory areas were disorganized in 
terms of the patients’ body representation. Because of that effect, patients tended to perceive 
the vibration stimuli on more distal body areas than the point upon which the stimuli were 
delivered. In Fig 2D we use a new graphical representation, which we refer to as the vibration 
map, to depict the evolution of patient P6’s recovery on vibration sensation. During the first 
evaluation (4‘° month) this patient reported no sensation in both left and right legs. Two 
months later, the patient started perceiving two regions on the right side (the ribs and the hip) 
and three on the left side (ribs to the knee). However, when asked to report the stimuli loca- 
tion, the patient reported feeling that the vibration applied to his hips felt like it originated in 
the knees, whereas the vibration on his left knee felt as if it was delivered to his left ankle. In 
other words, the patient perceived the stimulation more distal than the actual location to 
which it was applied. Several other instances of spatial localization errors were observed with 
this patient throughout the assessments. However, with training, the overall vibration map 
became more organized, and by the end of the protocol, this patient was able to correctly per- 
ceive vibration stimulation up to the knee on the right side and up to the foot on the left side. 

Other patients in GR1 also experienced similar errors related to the correct spatial localiza- 
tion of a vibratory stimulus. They included, in addition to locating the stimulus in a more distal 
joint, locating it in a more proximal joint or in a contralateral joint. In all cases, these errors 
were triggered by the expansion of the sensory recovery (Fig 3E) below the original SCI. Asa 
rule, patients often located the vibratory stimulus to more distal than proximal joints (39 cases 
vs. 5 respectively). The instances of misallocating the stimulus to a contralateral joint stayed 
constant over the training period (overall 36 cases). The highest incidence of these spatial 
localization errors took place between the 12" and the 16" month of training. After 22 months 
of training, and following a period in which the vibration score plateaued, the number of 
errors decreased. Consequently, the resulting vibration map became more organized. 


Motor function improvements 


Motor functions were tested following the standard AIS evaluation of five key muscles [6] (Fig 
4A). To have a complete picture, we assessed seven non-key lower-limb muscles (Fig 4B), 
three abdominal muscles and the anal sphincter (Fig 4C). 
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Fig 3. Proprioception and sensitivity to vibration. (A) Mean+SEM for proprioception (B) for vibration score for GR1 (black) and patient P7 
(red). (C) Eight stimulation areas for the vibration test. Patients were blindfolded during the exam and had to report if they could feel the 
vibration and report the location of the stimulation (rib, hip, knee, ankle or foot). (D) Vibration map for patient P6. A solid line connects the 
actual vibration position (left circle (R)ib, (H)ip, (K)nee, (A)nkle and (F)oot) and the felt vibration position (right circle) by the patient. A 
dashed line means that the patient indicated the contralateral leg. (E) Occurrences of confusion toward a proximal joint (e.g., stimulation was 
done in the knee and patient-reported he felt in the hip), distal joint or with a contralateral joint. 


https://doi.org/10.1371/journal.pone.0206464.g003 


Except for residual contraction in one muscle for patient P3 (motor score of 1 for extensor 
digitorium longus), all patient received a motor score of 0 for the all tested lower-limb muscles 
(12 per leg) at their admission (baseline measurement). This initial evaluation was confirmed 
by our medical team at the onset of the training, meaning that, before our training, none of 
our patients showed any voluntary motor activity in their lower limbs. Throughout the 28 
months of training (Fig 4A), we observed continuous motor improvement in the key muscles; 
with first visible contractions (motor score = 1) and first active movements (without gravity 
action, motor score = 2) appearing after 7 months of training. After 22 months of training 
patients, P1 reached a motor score of 3 (active movement against gravity) for the hip flexion 
(S2 Video). 

After 28 months of training, we found active movements (score of 2 and above) in 10 out of 
24 tested lower-limb muscles of patient P1, eight muscles for patient P3 and P8, four muscles 
for patient P4 and P5, and one for patient P6. We also observed motor function recovery in 
the abdominal muscles, in the three patients who had the highest lesions, namely P4, P5 and 
P6 (Fig 4C). Finally, in five patients we measured the presence of sphincter control, a muscle 
whose spinal motor roots originate in the most caudal part of the spinal cord (S4-S5). 

The lower extremity motor score (LEMS) was calculated (Fig 4D) by summing the scores 
for the lower-limb key muscles. The most prominent improvement was observed in the 
patients with more distal SCI locations (S2, S3 and S4 Videos), namely patients P1, P3 and P8 
(lesion range T10-T11, AIS A) (final score respectively 10, 12 and 11). Patients with a more 
proximal SCI began to show motor improvements after a longer training period: on the 10 
month for P2 (T4, AIS B), P4 and P5 (T7-T8, AIS A) and the 12" month for P6 (T4, AIS A). 
This observed difference in improvement rate among the patients was somewhat expected; the 
lower-limb muscles are innervated by lumbar and sacral roots. Therefore a patient with a T11 
lesion is closer to this level than a patient with a T8 lesion. 

As in the case of the sensory recovery, patient P7 displayed an above-average improvement 
in the motor score during the first year (Fig 4E, red line). However, after protocol discontinua- 
tion, he underwent a partial motor regression over the next 16 months, ending up at a level 
comparable to what he had achieved at the 7th month of training. 

Overall, the motor improvement observed in each one of our patients (ranging from 4 to 12 
points and, on average, 7.2 points by the end of the protocol) was highly significant when com- 
pared to the onset of training (0 for all), and with the spontaneous improvement rates reported 
in the literature (scores <1 point when the rehabilitation started a year after the injury [12]). 

In some cases, the lesion extended over several spinal levels (for example patient P8’s lesion 
is between T4 and L4). This raised an important question: are all the muscles that recovered 
activity innervated at the lesion level rather than below? In other words, could our observa- 
tions be explained by a mechanism of spontaneous recovery at the lesion level of the lower 
motor neurons (a mechanism known as root recovery [45,46])? To answer this, we calculated, 
for each patient, the distance between the muscle’s innervation root and the lowest part of the 
anatomical lesion. For example, patient P4’s MRI scan revealed a lesion at the T7-T9 level. As 
expected, at the onset of the training all myotomes under the lesion level were silent (Fig 5A). 
In clear contrast, by the end of the protocol, we observed that recovery extended up to seven 
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Fig 4. Motor improvement. Clinical evidence. (A) Five key lower-limb muscles (proximal and distal rectus femoris, tibialis anterior, 
extensor hallucis longus and gastrocnemius) were evaluated eight times throughout the training [6] (0 to 28 months) and before starting the 
training (B: baseline). The motor score describes the amplitude of a contraction from 0 (absence contraction) to 5 (normal contraction). (B) 
Motor score for seven non-key lower-limb muscles measured at the onset of the training (0) and the end of the end of the training (28 
months) for all patients. (C) Number of patients (over eight) with present muscle contraction for three abdominal muscles and the anal 
sphincter muscle (myotome $4-S5). (D) The LEMS is obtained for each patient by summing the score of all key muscles reported in panel A 
bilaterally. Missing data periods are in black. (E) Mean+ SEM of the motor score for GR1 patients is reported in black and motor score for 
patient P7 is in red. 


https://doi.org/10.1371/journal.pone.0206464.g004 


spinal levels below the original anatomical lesion (e.g., a level 2 contraction found in the glu- 
teus maximus muscle which is rooted in L4). 

We performed the same analysis for patients P2, P3, P5, P6 and P8 (Fig 5B). The anatomical 
lesion level (AL) for all these patients was obtained through analysis of MRIs (Table 1, S2 Fig). 
At the onset of the training, we detected very few cases of active myotomes below the AL. A 
very different result was observed 28 months later (at the protocol’s end), we observed multiple 
instances of voluntary contractions in muscles that are innervated by spinal nerves that origi- 
nate below the original AL. For example, for patient P2 we observed contraction in muscles 
that originated up to 13 spinal segments below the AL; similarly, for patient P3, we identified 
voluntary muscle contractions five segments below the AL, six for P5, 11 for P6 and two for P8 
(Fig 5B). Therefore, we found that the motor improvement in our patients was well below the 
original SC lesion level and could not be simply explained by a spontaneous root recovery 
mechanism. 

The occurrence of significant motor recovery in all patients was further corroborated 
through surface EMG recordings of voluntary contractions in multiple lower-limb muscles 
(Fig 5C). For example, in the case of the gluteus maximus (GMx, hip extensor, L5-S2), we 
observed a significant increase in contraction force in all patients. The first EMG measure- 
ments were obtained 7 months after the training onset. At this point, we did not observe signif- 
icant contractions in patients P2, P4, P5, P6, and P8. However, after 2 years of training, the 
same five patients regained the capacity to control GMx voluntarily: P2 and P8 activated GMx 
bilaterally; P4 and P5 consistently contracted their left GMx. Patient P6 performed less fre- 
quent contractions; they were nevertheless aligned with therapist instructions suggesting that 
the motor activations were voluntary. Patients P1 and P3 experienced the earliest signs of 
motor recovery; they began producing contractions of their previously paralyzed muscles 7 
months after the protocol onset, and contraction force strengthened with continued training. 
Patient P7 went from 0 to consistent left gluteus contractions between the 7" and 13" month 
of training. 


Visceral functions improvements 


Our patients also exhibited an expressive autonomic function recovery, which was represented 
by consistent improvements in intestinal, urinary and sexual function, as described in Tables 
3-5. Overall, five patients, whose condition changed from AIS A/B to C 28 months after train- 
ing onset, (P1, P2, P3, P4, P8) regained voluntary anal sphincter motor control (Table 3). 
Moreover, all seven patients recovered pain and deep pressure sensitivity at the last sacral der- 
matome, assessed using a pinprick test and the deep anal pressure evaluation, after 2 years of 
WA-NR training. Consequently, all of them significantly improved their ability to inhibit defe- 
cation voluntarily (Table 4). Four patients recovered the ability to experience anal sensation 
during defecation and, even more importantly for the patients’ quality of life, five regained 
awareness of the need for defecation. 
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Table 3. ASIA sacral evaluations. 


Fig 5. Motor improvement. Neurological and neurophysiological evidence. (A) Example of active myotomes compared to the 
Anatomical Lesion (AL) shown for patient P4. For each muscle, the graphic shows the corresponding myotome level (considering the 
principal nerve root, S2 Table), and the clinical score at the onset (0) and the end (+28 months) of the WARN training. The MRI of this 
patient’s spinal cord revealed an AL extending between T7 and T9 segments. Accordingly, at the onset of the training, the patient had 
preserved motor functions in the upper-limbs and the upper abdomen muscle (spinal nerve roots are located at T7-TS8 level), but could 
not contract the middle and lower abdomen muscles (T9-T12 segment) nor any of the lower limb muscles. After 28 months of the 
WANR training, this patient had recovered partial motor functions in the middle and lower abdomen as well as in multiple lower limb 
muscle innervated under the AL, namely, rectus femoris proximal (L2) and distal (L3), hip adductor (L2), gluteus medius/maximus, 
tibialis anterior (L4), and gastrocnemius (S1). B) For each patient, we considered the muscles where motor functions were clinically 
observed (ASIA motor score = >1) and calculated the distance to the AL. The distance was calculated as the number of myotomes 
between the spinal nerve root of the muscle and the lowest segment of the AL. A positive value in the graph, corresponds to a muscle 
that is rooted below the anatomical lesion; and negative values refer to muscles rooted above the lesion. We report results for the onset 
(0) and the end of the training (28). Trunk muscles are reported with an open circle, lower limb muscles with an open triangle and 
upper limb muscles are not considered. (C) EMG envelops for the gluteus maximums muscle for all patients. Patients were instructed to 
contract their legs for periods of 5 seconds over a 3-minute period. Verbal instructions were given to the patient by the PT; and 
instruction periods are shown in gray in the graph. A dark gray area highlights the trials where the patient had a significant GMx 
contraction (> mean + 3xSD of the baseline), and light gray indicates those where the contraction did not reach significance. Muscle 
responses are shown for all patients (P1 to P8) at an early stage and later in training. 


https://doi.org/10.1371/journal.pone.0206464.g005 


Four patients became capable of voluntarily inhibiting urination (Table 5), while six sub- 
jects recovered their ability to perceive the need for bladder emptying and five to perceive the 
catheter during bladder emptying. Due to these improvements, these patients no longer had to 
rely on indirect clinical signs, such as sweating, tachycardia and increased spasticity to sense 
the need to empty their bladders. Such an improvement in urinary functions possibly contrib- 
uted to the reduction of urinary infections. Indeed, one patient (P6), who experienced repeated 
cases of lower tract urinary infections before the training onset, exhibited a marked decrease in 
these events over time: considering the 28 month time range of our protocol, this patient expe- 
rienced four episodes of lower tract urinary infection during the first year, two during the sec- 
ond year and none during the last 6 months of training. We also observed improvements in 
sexual and genital function in both female and male patients (Table 5). Both female patients 
recovered sensitivity during sexual intercourse, as well as awareness and sensitivity to men- 
struation flow and cramps. While the reflexive erection (controlled by parasympathetic centers 
S2-S4) was preserved in all male subjects, three of them started experiencing psychogenic erec- 
tions (physiologically controlled by supraspinal sympathetic centers and thoracolumbar sym- 
pathetic spine center at T11-L2), sensitivity during intercourse and ejaculations (conveyed by 
synergism between supraspinal and spinal centers T11-L2 and S2-S4) functions. 

Overall, improvement in bowel, bladder, and sexual function suggested that patients experi- 
enced some degree of neurological recovery at the level of sacral segments (S2-S5), which was 
significantly below the patients’ original SCI. 


Sacral (ASIA) 0 4 7 10 12 16 22 28 

Motor 
Voluntary anal contraction (S4-S5 myotomes) 0 0 0 1 2 3 4 5 
Sensitivity 

Deep anal pressure evaluation (S4-S5 dermatomes) 1 1 1 3 4 4 6 7 

Pinprick pain sensitivity (S4-S5 dermatomes) 1 1 1 3 4 4 5 7 

Reflex 

Pinprick reflex evaluation (S4-S5) 6 6 6 6 6 6 6 7 


One motor, two sensory and one reflex tests about the last sacral dermatome/myotome function. Tests measured directly by a physician [6]. Results are shown for eight 


measurements done throughout the training period (at training onset and after 0, 4, 7, 10, 16, 22, and 28 months) for GR1 patients (n = 7). 


https://doi.org/10.1371/journal.pone.0206464.t003 
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Table 5. Genitourinary evaluations. 


Table 4. Intestinal function evaluation. 


Intestinal 0 4 z 10 12 16 22 28 
Motor 

Ability to voluntary inhibit defecation 0 0 0 0 1 1 2 

Fecal incontinence 1 1 1 1 0 0 0 0 
Sensitivity 

During feces elimination (defecation) 1 2 2 2 2 2 4 
Awareness of the need for bowel emptying 1 1 1 1 2 2 


Two motor and two sensory tests about the bowel/intestinal functioning for GR1 (n = 7). Tests were collected 
through a self-questionnaire based on ISCOS data sets [40,41]. 


https://doi.org/10.1371/journal.pone.0206464 1004 


Principal factors for sensory and motor recovery 


Having documented a marked sensory-motor recovery in our patients, we next investigated 
which aspects of the WA-NR training protocol (TR) and which external factors (EX) could 
best account for these partial clinical improvements. 

Improvement in nociception in our patients was significantly correlated with BMI training 
hours (Fig 6A, R = 0.29, P = 0.03, Pearson coefficient of correlation), whereas correlation with 
TR-LOC hours (R = 0.23), did not reach significancy (P = 0.08). Thus, the improvement rate 
in nociception was most marked during a period with increased BMI-based training. The 
same was true for the improvement rates for crude touch sensation (R = 0.44, P = 0.002 for 
TR-BMI, R = 0.17 for TR-LOC, n.s), for pressure improvement (R = 0.43, P = 0.006, versus 
R= 0.04, n.s.), for proprioception (R = 0.33, P = 0.04 versus R = 0.07, n.s.) as well as for motor 
improvements (R = 0.34, P = 0.01, for TR-BMI, and 0.17, n.s., for TR-LOC). 

Factors like the patients’ age and time since their SCI had no influence on the improvement 
rate in any of the measured sensory-motor metrics. None of the external factors were corre- 
lated with the improvements. 


Urinary 0 4 7 10 12 16 22 28 
Motor 
Ability to voluntary avoid urination 0 0 0 1 4 
Involuntary urine leakage 5 6 6 6 3 
Sensitivity 
Awareness of the need for bladder emptying 1 2 1 1 1 1 6 
During bladder emptying with catheter 1 1 1 1 1 1 5 
Sexual, Genital 0 4 7 10 12 16 22 28 
Female 
Sensitivity: during sexual intercourse 0 1 1 2. 2 
Sensitivity: Menstruation awareness 0 0 2 
Male 
Sensitivity: During sexual intercourse 1 1 1 1 1 1 1 2 
Motor: psychogenic erection 0 0 0 0 0 0 0 2 
Motor: Reflex 5 5 5 5 5 5 5 5 
Motor: Ejaculation 0 0 0 0 0 0 0 2 


Responses to the questionnaire [40,41] related to urinary functions are divided into motor and sensitivity aspects for all patients (n = 7). Questions related to sexual 


aspects are divided between female (two participants) and male (five participants) patients. 


https://doi.org/10.1371/journal.pone.0206464.t005 
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A. Correlation between improvement 
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Fig 6. Neurological improvement and correlation with training the protocol. (A) The coefficient of correlation between sensory 
and motor neurological improvements and the number of training hours of active locomotion, BMI-based training, patients SCI 
height, time since lesion and patients ‘age. (B) AIS grade improvement at 28 months. Patient P7 stopped the training after 12 months. 
A follow-up measurement was done with this patient, 16 months after he stopped the training. (C) WHOQOL-BREF [36] score for 
four subdomains. 


https://doi.org/10.1371/journal.pone.0206464.g006 


Altogether, all seven patients who remained enrolled in the WA-NR protocol for 28 months 
changed their initial AIS classification (Fig 6B, see details in S1 Fig). Patients P1, P3, and P8, 
who had the lowest lesions (T10 -T11), and P2 (AIS B at admission), changed their AIS grade 
during the first year. Patients P4, P5 and P6, who had higher lesions (T4-T8), changed AIS 
grade during the second year of training. P7 (T5/T7) was the only patient that stagnated; possi- 
bly because he discontinued the training early. 


Quality of life and neuropathic pain 


To measure the patients’ quality of life (QoL), the WHOQOL-BREF (World Health Organiza- 
tion Quality of Life Assessment Instrument-Bref) questionnaire was applied six times (first 
time after 7 months of training) with GR1 patients. Total and partial scores, divided into four 
domains (physical, psychological, social and environmental) are exhibited in Fig 6C. We 
observed a significant increase in the physical domain between the 7th (76.0£4.5, meantSEM) 
and 22nd months (82.1+3.9) of training (t-test, P<0.05). Similar to the clinical motor score, 
the WHOQOL physical domain score decreased during the break period between the 10th 
and 12th months of the protocol (Table 2). In the psychology domain, we found a significant 
improvement throughout the training (from 80.9+3.38 to 88.13+3.7, considering the seventh 
and 28"" month, t-test, P<0.05). We, however, did not see any significant changes for the social 
(80.9+3.9 to 84.5+4.6, P>0.1) and environment (68.343.9 to 68.7+7.3, P>0.1) domains, the 
two aspects on which our protocol did not focus. 

The details for the physical domain are shown in Table 6. Following training, patients 
reported better ability in performing their activities of daily living; as well as their capacity for 
working. This suggests that the training with the WA-NR helped patients reacquire confidence 
in their ability to work and to achieve higher goals in their lives. 

The score for pain and discomfort started and remained positive over the 2 years of the 
training period; patients did not report ‘that physical pain prevents [them] from doing what 
[they] need to do.’ For a more detailed measurement of this aspect, the McGill standard evalu- 
ation was also applied (Table 7). We considered only the cases of neuropathic (or more pre- 
cisely, myelopathic) pain like burning/tingling, shocks perceived in areas at the level and 
under the lesion) and discarded those that were related to external factors (headaches, a pos- 
tural pain above the lesion level unrelated to the training, etc.). Patient P4, P5, P6, and P7 only 
presented a few cases of light neuropathic pain (McGill score of 1). Patients P1 and P8 reported 
a moderate and severe case of pain, respectively, at the onset of the training. The pain reported 
by P8 was especially uncomfortable, describing it as a sensation of burning in the feet; P1 
described moderate pain similar to the sensation of shocks in the leg. The perceived pain atten- 
uated for both patients after following the WANR training, downgrading to 0 for P1 and 1 for 
P8 (light pain). In one case an increase of neuropathic pain was observed after the onset of the 
training. Following partial neurological recovery, patient P3 reported a moderate pain in the 
right thigh. 

We found that the patients’ level of fatigue and energy stayed constant during all periods of 
training and only decreased during the break period (between the 10" and the 12 months); 
possibly because the WA-NR promoted routine physical training for our patients. Our training 
did not positively or negatively influence the patients’ sleep. The mobility item measured 
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Table 6. Mean + SEM score for all physical domains of the WHOQOL-BREE [36] questionnaire. 


WHOQOL physical domains 7 10 12 16 22 28 DIFF (28-7) 
Pain and discomfort 92.9+4.6 96.443.6 89.347.4 89.3+7.4 96.4+3.6 92.9+7.1 0.0 
Energy and fatigue 85.745.1 82.1+7.1 64.347.4 82.147.1 82.147.1 89.345.1 3.6 
Sleep and rest 75.0£7.7 71.4+6.5 78.6+6.5 71.4+8.5 78.6£6.5 71.4410.1 -3.6 
Mobility 82.144.6 75.0+0.0 71.443.6 71.443.6 78.6£3.6 78.6+6.5 -3.6 
Activities of daily living 71.4+£6.5 78.643.6 71.448.5 75.0£7.7 82.144.6 85.745.1 14.3 
Dependence on medication 64.3414.3 82.1+10.5 78.646.5 78.6£10.1 78.6411.5 67.9411.8 3.6 
Work capacity 71.448.5 TEAORE SS) 67.9+7.1 67.9+7.1 78.643.6 78.6£3.6 Fok 


The questionnaire was done respectively after 7,10, 12, 16, 22 and 28 months of training. In the last column, we report the difference between the mean score after 28 


months of training and the mean score of the first assessment. 


https://doi.org/10.1371/journal.pone.0206464 1006 


Table 7. McGill score. 


patients’ autonomy and accessibility to perform their daily life activities (e.g., indoor accessibil- 
ity.); the item ‘dependence on medication’ measured patients general use of medication (con- 
sidering both chronic and acute pain). As expected, our training did not influence these two 


items. 


Concerning the psychological domain, we documented an improvement in five out of the 
six sub-items (Table 8). Patients’ reported enjoying their lives more (an increase of 7.1 for posi- 
tive feelings), corroborated with a decrease in the occurrence of negative feelings, such as blue 
mood, despair, anxiety, and depression (improvement of 10.7 points). Patients reported being 
more satisfied with themselves (+7.1 point in self-esteem) reaching a very high score by the 
end of the training (96.4%). We believe that the patients’ improvement in the thinking/ 


Pl P2 P3 P4 PS P6 P7 P8 

Months of training | 0 28 0 28 0 28 0 28 0 28 0 28 0 12 0 28 
Throbbing | 0 1 0 0 1 1 0 2 0 0 1 0 1 1 3 0 
Shooting 2 0 0 1 0 0 0 0 0 0 0 1 1 1 0 0 
Stabbing 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 

Sharp 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
Cramping| 0 0 0 0 0 0 2 1 0 0 0 0 0 0 1 0 
Gnawing 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 
Hot-burning| 0 0 0 0 0 0 0 1 1 0 0 1 0 1 3 1 
Aching 0 0 0 0 1 2 1 0 0 0 0 0 0 0 1 2 

Heavy 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 

Tender 0 0 0 0 0 2 0 1 0 0 0 0 0 0 1 2 

Splitting 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
Tiring-exhausting| 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 
Sickening 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 0 

Fearful 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
Punishing-cruel| 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
Sum 2 1 1 1 2 5 3 5 1 0 2 2 3 4 14 5 

Sum neuropathic | 2 1 1 1 1 5 0 1 1 0 2 1 2 2 3 1 


Detail for the McGill score considering the descriptors for two evaluations, at the onset (0) and at the end of the training (after 12 months for P7 and after 28 months for 


the other patients). Cases where the descriptors of the pain are below the lesion are considered as neuropathic pain and underlined, those above the lesion (as postural 


pain, headache, etc.) are not considered to be due to the spinal lesion and therefore not considered for the current analysis. 


https://doi.org/10.1371/journal.pone.0206464.t007 
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concentration/memory item reflected the fact that training with the BMI specifically required 
subjects to focus on their rehabilitation tasks. Several studies have shown that BMI training is 
beneficial for improving focus and concentration [47]. Also, we observed that body-image pla- 
teaued at a high score throughout the training (~90%). Finally, one of the most interesting 
results came from asking whether patients ‘[..]feel [their] life to be meaningful’ (spiritual/per- 
sonal beliefs). We observed a continuous improvement over the training period, suggesting 
that the participation in the WA-NR protocol had a positive impact on the patients’ self- 
esteem. 


Discussion 


The present study reports a systematic and unprecedented partial neurological recovery in 
patients diagnosed with chronic complete (AIS A) and motor complete (AIS B) paraplegia, fol- 
lowing long-term non-invasive neurorehabilitation [33]. Based on a study that gathered clini- 
cal data from a group of eight SCI patients over a 28-month period, we found that the longer 
the patients trained under the protocol that combined a BMI, visuo-tactile feedback and active 
locomotion, the larger was the sensory-motor and visceral recovery observed below the SCI 
level. This was true for both somatosensory (tactile, nociceptive, proprioceptive, pressure and 
vibration), motor (voluntary contraction observed for multiple myotomes below the original 
lesion), and autonomic functions (sexual, intestinal and urinary). 

At the core of the WA-NR protocol, concurrent BMI-based control of virtual and mechani- 
cal actuators combined brain activation with continuous visuo-tactile feedback and physical 
training, assisted by a body weight support system and robotic gait therapy devices. Thus, the 
principal difference between the WA-NR protocol and other existing neurorehabilitation para- 
digms is that it focuses neither on the physical, nor on the BMI approach per se, but instead it 
creates the conditions to simultaneously engage both cortical and peripheral signals that con- 
verge towards the level of the SCI. As such, the intended goal is to reinforce the potential phys- 
iological role played by spinal tracts that have survived the original injury. 

Depending on the cause, a traumatic SCI can lead to a variety of lesions, generated by con- 
tusion, compression or penetration, which is followed by massive necrosis of affected neural 
circuits during the first 18 hours post-injury [48]. During the subsequent weeks, the neuroim- 
munological system deflagrates a cascade of mechanisms that may expand the lesion above 
and below its original epicenter. Cavity or cyst formation and demyelination may also occur, 
damaging both ascending and descending pathways [48]. 

Previous studies have shown that 84% of clinically complete paraplegics (AIS A) exhibit 
some neurophysiological activity below the level of the injury [7,49] and are therefore referred 
to as “discomplete” SCI [8]; post-mortem analysis by Kakulas et al. [9] confirmed the presence 
of fibers in patients diagnosed as having a clinically complete injury. Confirming this assess- 
ment, MRI scans of our patients showed the residual spinal cord continuity in three out of the 
six patients (two AIS A, one AIS B). 

Based on these new observations, we suggest that even a small portion of surviving spinal 
cord axons can contribute to a meaningful clinical and functional recovery, provided that they 
can be properly re-engaged by a long-term rehabilitation protocol like ours. In support of this 
point of view, animal models of SCI have shown that sparing of about 10-15% of the spinal 
cord is sufficient to support a partial recovery of locomotion [50]. In human subjects, the exact 
amount of spared SC white matter needed to observe similar levels of functional neurological 
recovery remains unknown. However, in a study in which a cordotomy was performed in 44 
SCI patients [51] to alleviate secondary cancer-induced pain, Nathan et al. showed that even a 
complete bilateral section of the anterior portion of the spinal cord, containing motor tracts, 
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Table 8. Mean + SEM score for all psychological domains of the WHOQOL-BREF [36] questionnaire. 


WHOQOL psychology domains Z 10 12 16 22 28 | DIFF (28-7) 
Positive feelings 71.4+3.6 78.6+3.6 75.0£5.5 75.0£5.5 82.1+4.6 78.6£6.5 | 7.1 
Self-esteem 89.3+5.1 85.7+5.1 89.3+5.1 | 89.3+5.1 96.4+3.6 96.4+3.6 | 7.1 
Thinking, memory and concentration 82.1+7.1 82.144.6 82.1+4.6 78.64£6.5 89.3+5.1 89.3+5.1 Til 
Bodily image and appearance 89.3+5.1 89.3+5.1 85.7+5.1 92.9+4.6 92.9+4.6 89.3+5.1 0.0 
Negative feelings* 78.6£3.6 85.7+5.1 85.7+£5.1 89.3+5.1 | 92.9+4.6 89.3+5.1 | 10.7 
Spirituality/personal beliefs 75.0£5.5 78.6+3.6 78.646.5 78.648.5 82.1+4.6 85.7+5.1 | 10.7 


The questionnaire was done respectively after 7,10, 12, 16, 22 and 28 months of training. The difference of mean score after 28 months as compared to the mean score of 


the first assessment. 


* High scores mean fewer negative feelings. 


https://doi.org/10.1371/journal.pone.0206464.t008 


did not significantly affect patients’ motor functions. They suggested that the motor tracts in 
the posterior half of the spinal cord could compensate for the functions originally mediated by 
the anterior portion. 

Our studies confirm the role played by the employment of BMI-based training to induce 
both partial sensory and motor recoveries. We found that periods of increased hours of BMI- 
based training yielded the most pronounced clinical improvements. Importantly, we also 
observed that the improvement rate was dependent neither on the period since the patients’ 
original SCI, nor the patients’ age. 

But what mechanism could account for this recovery? We hypothesize that by triggering an 
extensive process of cortical and spinal cord functional plasticity, our BMI paradigm created 
the conditions for our patients to recover sensory-motor and visceral functions. These results 
are in accordance with recent observations in a rat model [27,31]. Moreover, our results indi- 
cate that both neurological and local muscular factors contribute to the final motor outcome 
following the WA-NR protocol. Indications that recovery is happening due to neurological 
factors include a proximal-to-distal order of recovery. The L2 -innervated muscles had volun- 
tary contractions in all eight patients, while more distal levels like L5 were seen in two patients 
and S1 in one patient only. If there were only neurological factors involved in the final out- 
come, it should be expected that all muscles innervated by a nerve root would respond equally, 
which is not the case. The analysis of L4-innervated muscles allows us to compare the tibialis 
anterior (key-muscle) with five other muscles (gluteus maximus, gluteus medius, medial ham- 
string, lateral hamstring, and extensor digitus longus) which differ in size and location. The 
glutei have shown better scores among the majority of patients, while hamstrings and extensor 
digitus longus had poor responses. Tibialis anterior, which is a key-muscle for L4 evaluation 
had lower scores than the glutei, showing that non-key muscle testing is indeed important to 
rule out inhomogeneities of recovery due to local muscular factors. 

Previously, BMI-evoked cortical plasticity has been reported in the rehabilitation of stroke 
patients (36). These studies have shown a considerable clinical effect of BMI training even in 
patients with severe neurological impairment. Similar to the classical physical therapy proto- 
cols for incomplete SCI [52], repeated active motor tasks, which promote activity-dependent 
rehabilitation, have been recognized for inducing partial motor recovery in stroke patients 
[53,54]. Constraint-induced movement therapy, for example, has been successfully used for 
stroke rehabilitation, even at the chronic phase [55]. However, this approach relies on the utili- 
zation of the patient’s residual motor functions, a condition that is absent in 30-50% of stroke 
victims [56]. Therefore, BMI-based therapy was adopted for these most severe stroke cases, in 
which no residual motor function was present. This method aimed at rehearsing a lost motor 
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function, through real-time decoding of the patient’s motor imagery and online feedback 
(visual or proprioceptive [57,58]). In this approach, the BMI serves as a cortical operant condi- 
tioning (as shown in pioneer work of Fetz and collaborators [59]). We believe that a similar 
neurophysiological mechanism was activated in our SCI patients when they were subjected to 
the WA-NR protocol. At the onset of the training, our patients, who were diagnosed with com- 
plete paraplegia, could not follow classical training with the active motor task. Instead, our 
BMI-based neurorehabilitation, which integrated decoding of cortical motor imagery with 
visuo-tactile feedback, provided the driving force for triggering the kind of cortical plastic pro- 
cess that mediated their recovery. 

Concomitant to a partial somatosensory and motor recovery, we also demonstrated that 
training with the WA-NR protocol induced a significant change in our patients’ perception of 
their own bodies, as evident from the partial recovery of tactile, proprioceptive and vibratory 
sensitivity. Thus, our protocol affected the patients’ body schema, which involves the multisen- 
sory integration of visual, kinesthetic and proprioceptive information to provide humans with 
a sense of being and existing in space [60,61]. Previous studies have shown that the body 
schema is plastic and, hence, can become distorted after SCI [62,63] or a limb amputation 
[64]. At training onset, patients did not detect the presence of vibration stimulation in their 
lower limbs. When they first began perceiving tactile sensations from the parts of their body 
that had remained numb since their original SCI (i.e., for many years), their perception of the 
tactile stimuli location was distorted. As a rule, after the patients began to regain sensations in 
their legs, they tended to perceive vibration in more distal parts of the leg, compared to the 
actual stimulation site. 

One explanation of this effect is the sensory stimulation activated not only the cortical areas 
representing the stimulated body part, but it also invaded adjacent representations of the more 
distal parts that had remained silent for a long time. Such “filling-in” of sensory deprived areas 
of the primary somatosensory cortex has been previously reported in animal models [65] and 
amputee patients [66,67]. But, the observation of a constant number of misplacement errors 
toward the opposite leg, an area not adjacent in the cortical somatotopic representation, sug- 
gests additional mechanisms to the classic ‘invading’ representation explanation. A possible 
explanation is that the peripheral information entered the somatotopic representation of the 
stimulated body part but was interpreted incorrectly because of the body schema distortion 
caused by many years of sensory deprivation. 

Following the sensory recovery in the lower-limbs, we observed that at first the patients 
could detect the presence of stimulation but their spatial representation was distorted, and 
then, the sensory map became more organized. We believe that the extensive training using 
congruent visual, tactile and proprioceptive signals, was essential for patients to recover a 
more structured body representation. Indeed, we have demonstrated in a previous study that 
the congruent use of the virtual reality and tactile feedback setup used in the WA-NR induced 
a significant change of body schema representation in SCI patients [35], in coherence with 
numerous studies showing this effect with healthy subjects [68,69]. Overall, we propose that 
the temporal evolution of our patients’ perception of their bodies emerged from a complex 
process of activity-dependent plasticity, occurring in the body representations that exist at the 
cortical level, which altogether define the patient’s body schema [60]. Likely, this cortical plas- 
ticity was paralleled by a similar process taking place at subcortical levels, both of which were 
influenced by peripheral contributions to this body representation. In addition to lower limb 
sensory-motor dysfunction, the most devastating effects of SCI are genitourinary, gastrointes- 
tinal and sexual dysfunction [70]. Surveys have shown that paraplegic patients ranked as their 
first priority the recovery of sexual functions (26%), followed by bladder/bowel functions 
(18%), before movement and sensation recovery (16% and 7.5%) [71]. In our study, five out of 


PLOS ONE | hitps://doi.org/10.1371/journal.pone.0206464 November 29, 2018 25/33 


©-PLOS | ove 


91 


Sensory-motor, visceral and psychological improvement in paraplegics 


the seven patients became aware of the need for bowel emptying and recovered the ability to 
inhibit voluntary defecation 22-28 months after the training onset. In addition, four patients 
regained the ability to inhibit urination and six recovered awareness of the need for bladder 
emptying. These changes had a very positive impact on the patients’ health, as they reduced. 
the risk of patients acquiring a urinary infection and exposure to autonomic dysreflexia that 
could lead to uncontrolled hypertension. Significant gains in the patients’ ability to inhibit uri- 
nation also gave them more flexibility in their daily activities and improved the social 
interactions. 

Another fundamental observation of this long-term study was that four patients (two males 
and two females) partially recovered their sexual functions. This observation suggests that the 
partial neurological recovery obtained was not constrained to the lower-limb somatosensory 
and motor functions, which were the primary focus of our BMI and the physical training, but, 
instead, resulted from a generalized neurological recovery, which also manifested itself as an 
improvement in major visceral functions. 

Although the rate of clinical improvement of the patients was higher during the first year of 
training compared to the second, the patients’ recovery continued to improve for the 28 
months of training. Yet, it is not entirely clear at this point to what extent the effects of 
WA-NR are preserved after a patient stops this training. The results for one patient showed 
that, after displaying promising clinical improvements, above the group average, during the 
first 12 months of training, this patient maintained the majority of his sensory gains for the 
remaining 18 months, even after he discontinued training. This was true for the rate of 
improvement for tactile, nociceptive, proprioception and pressure sensitivity modalities, but 
not for vibration. Moreover, upon discontinuation, the same patient exhibited a clear regres- 
sion of motor functions gained during the first year of training but did not return to his start- 
ing point (which was equal to zero at the protocol onset). These observations indicate the 
importance of continued engagement with the WA-NR protocol. Longer follow-up periods 
with more patients will be needed to evaluate the long-term effects of discontinued training on 
the partial gains in sensory, motor, and visceral functions obtained with the WA-NR protocol. 

During our research, the patients had integral psychological support with the purpose of 
guiding expectations adjustments, regarding the neurological improvements that the patients 
could potentially exhibit. Our major concern was the impact of the walking training on 
patients, as subjects were previously prepared to independently perform activities of daily liv- 
ing (ADL) in a wheelchair and were already adapted to this condition. An important question 
that was raised in this study was how the training would impact the patients’ own subjectivity 
and change (if it does) their quality of life (QoL) perception. QoL is not merely the absence of 
disease, but the state of complete physical, psychological and social wellbeing. In recent years, 
QoL improvement for SCI has become a rehabilitation goal [72,73], and its assessment is con- 
sidered beneficial for evaluating the outcome of a multidisciplinary rehabilitation team 
approach. The patient’s self-perception of the QoL is considered an efficacy measurement of 
treatment and can also be used to evaluate the cost-effectiveness of interventions, and research 
and rehabilitation programs [74]. In our study, we observed increases in total, physical and 
psychological subdomains of QoL following our protocol. The physical domain exhibited fast 
improvement at the onset of the training, and a decrease during the break period, stressing the 
importance of maintaining continuous rehabilitation activities in these patients. 

Unlike changes observed in the physical domain, the psychological domain exhibited a 
gradual improvement and a long-term effect on the course of training with the WA-NR proto- 
col. While partial neurological recovery can contribute to an improvement in QoL in spinal 
cord injury patients [75], we suggest that another important factor can explain the positive 
effect observed during the execution of the WA-NR protocol: patients were fully engaged in 
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their training as they were encouraged to imagine themselves performing movements while 
they received rich visual, vibrotactile and proprioceptive feedback; in other words, patients 
were required to be active protagonists of their physical therapy and neurorehabilitation 
process. 

As part of our follow-up assessing the patients’ quality of life, we also controlled for 
instances of neuropathic pain. Below-lesion neuropathic pain (NP) is present in 34% of SCI 
patients [76] and can significantly reduce subjects’ quality of life and compromise the neuror- 
ehabilitation process [77]. In our case, as confirmed by WHOQOL results, we did not observe 
cases of NP that prevented patients from performing their training normally. However, in one 
case a severe NP reported at the onset of the protocol, and described as a sensation of burning 
in the feet, was later alleviated. We propose here a hypothesis for this mechanism of reduction 
of NP. Motor cortex (M1) is thought to play an important role in modulation of pain [78-80]; 
indeed stimulation of M1 is a well-studied treatment for chronic NP especially after deafferen- 
tation (post-stroke pain, brachial plexus avulsion, phantom limb pain and also post spinal cord 
injury NP). By promoting higher activation in the sensory-motor cortical areas, the BMI may 
have played an important role in the reduction of NP. Indeed, in a case study with one chronic 
SCI patient, 4 months of training with a non-invasive BMI was shown to promote a reduction 
in neuropathic pain [81]). Further studies with a group of patients with higher levels of neuro- 
pathic pain are necessary to investigate this hypothesis in a more comprehensive way. 

Overall, our patients’ training with the WA-NR promoted a sense of empowerment [82] 
and acted by strengthening the sense of competence, self-worth, and self-esteem of our 
patients. It allowed the individual to overcome the situation of helplessness and develop con- 
trol over their own lives. The concept of patients’ empowerment has been gathering interest in 
various health care domains [83-88] since it promotes the concept of self-determination of 
patients as agents of their health and healthcare [89]. We suggest that the empowerment com- 
ponent may have influenced the perception of our patients’ capacity to contribute to their own 
recovery and, consequently, help improve their QoL. This process generates a sense of auton- 
omy for patients, as they realized that they were actively involved in seeking some degree of 
clinical improvement. This improvement resonated in patients’ daily habits, as, for example, 
their dressing habit; for the first time since the lesion, two patients started using shorter cloth 
(skirts/shorts) revealing parts of their body they had been hiding. 

Overall, our long-term clinical results open new therapeutic perspectives for the rehabilita- 
tion of the most severe cases of SCI (AIS A and B), even at the chronic phase, while using a 
purely non-invasive BMI-based approach. Therefore, our findings suggest that existing or 
future technologies, created for incomplete AIS C patients, may also be used for patients origi- 
nally classified as AIS A/B. For example, a large number of existing orthoses and exoskeletons, 
which require lower limb EMGs for their actuation (see [90] for a review), could now be 
potentially considered for use with AIS A patients, following a period of BMI-based training, 
using a protocol like the WA-NR. Our findings also indicate that, given a small fraction of 
spared spinal cord white matter, a much larger than expected population of AIS A/B patients 
might benefit from neurorehabilitation protocols that actively engage the patients’ mental and 
physical activity, while providing them with rich visual and tactile feedback. 


Supporting information 


$1 Fig. ASIA score sheet for all patients at onset and at the end of the training. 
(PDF) 


S2 Fig. MRI of patients’ spinal cord. (A) MRI cuts of sagittal and (B) axial planes (T2 
sequence) at SCT level for patients P2, P3, P4. Myelomalacia (hyperintense signal, 7mm length) 
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is visible for patient P2 at the level of thoracic vertebra T2, and continuity of neural fibers are 
visible at the lesion level. For patient P3, we observed the spinal cord injury extending between 
thoracic vertebras T10 and L1 with the presence of remaining fibers (dark gray). T12 axial 
plane for the same patient reveals injury arachnoiditis (inflammation of spinal meninges) and 
fiber continuity (better visualization). For patient P4, close to the vertebral body fracture at 
T8-T9 level, we also observe neural fiber continuity; axial plane reveals dural sac septations at 
injury level and fiber continuity. (C) Example of 3D segmentation and projection on sagittal 
and coronal plans for patient P4, confirms spinal cord continuity at lesion level. 

(PDE) 


S1 Video. 3D MRI reconstruction. Three-dimensional reconstruction based on FIESTA 
sequence images for patient P4. The rendering was done with OsiriX Lite software. Published 
with permission of Associa¢géo Alberto Santos Dumont para Apoio a Pesquisa (AASDAP), Sao 
Paulo, Brazil. 

(MP4) 


$2 Video. Motor examination in a suspended position for patient P1. The motor exam was 
done 9 and 22 months after the onset of the training. The patient is instructed to flex the right 
hip. Published with permission of Associacéo Alberto Santos Dumont para Apoio a Pesquisa 
(AASDAP), Sao Paulo, Brazil. 

(MP4) 


S3 Video. ASIA motor examination of patient P3. The motor exam is done 32 months after 
the onset of the training. The patient is asked to align the lower limbs, performing hip adduc- 
tion and knee extension, for the left side and later for the right side. Published with permission 
of Associacéo Alberto Santos Dumont para Apoio a Pesquisa (AASDAP), Sao Paulo, Brazil. 
(MP4) 


$4 Video. Motor examination in a suspended position for patient P8. The motor exam was 
done 29 months after the training onset. The patient was instructed to move both legs alterna- 
tively backward (right and later left side) and forward (right and later left side); with 75-80% 
of body weight support during forward movement and 65-70% during backward movement. 
Is possible to see EMGs electrodes placed at the lower limbs, for neurophysiology analysis, and 
the hands of a therapist are stabilizing the pelvis, avoiding body rotations. Published with per- 
mission of Associa¢géo Alberto Santos Dumont para Apoio a Pesquisa (AASDAP), Sao Paulo, 
Brazil. 

(MP4) 


$1 Table. WHOQOL-BREF set of questions per domain. 
(DOCX) 


$2 Table. For each muscle, the corresponding nerve, nerve root range. The reported princi- 
pal nerve is the one reported in ASIA assessment (except extensor digitorum longus which is 
not part of the ASIA assessment). 

(DOCX) 


S3 Table. Patient’s proprioception score per movement after 28 months of training. 
(DOCX) 
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Spinal cord injury (SCI) impairs the flow of sensory and motor signals between the brain and the areas 
of the body located below the lesion level. Here, we describe a neurorehabilitation setup combining 
several approaches that were shown to have a positive effect in patients with SCI: gait training by means 

:_ of non-invasive, surface functional electrical stimulation (sFES) of the lower-limbs, proprioceptive and 

: tactile feedback, balance control through overground walking and cue-based decoding of cortical motor 
commands using a brain-machine interface (BMI). The central component of this new approach was the 
development of a novel muscle stimulation paradigm for step generation using 16 sFES channels taking 
all sub-phases of physiological gait into account. We also developed a new BMI protocol to identify left 
and right leg motor imagery that was used to trigger an sFES-generated step movement. Our system 
was tested and validated with two patients with chronic paraplegia. These patients were able to walk 
safely with 65-70% body weight support, accumulating a total of 4,580 steps with this setup. We 
observed cardiovascular improvements and less dependency on walking assistance, but also partial 
neurological recovery in both patients, with substantial rates of motor improvement for one of them. 


Every year, about 250,000-500,000 people worldwide suffer a spinal cord injury (SCI) as a result of traffic acci- 
dents, falls, other traumatic accidents or violence’. SCI leads to severe impairments of the sensory, motor and 
autonomic functions below the lesion level, as well as secondary clinical conditions such as pressure ulcers, uri- 
nary tract infections, and osteoporosis. In addition to the gravity of these clinical effects, SCI incurs substantial 
financial costs, both for the individual and society. For patients diagnosed with the most severe cases of SCI 
(patients presenting no motor functions below the lesion (AIS B) and patients presenting neither motor nor sen- 
sory functions (AIS A)), at the chronic phase of the lesion (a year after the injury), the chances for a spontaneous 
recovery are negligible**. Up to now, there is no systematic approach to restoring neurological functions in such 
patients. 
Over the past decades, a variety of new approaches for SCI rehabilitation have been introduced*». These efforts 
: aimed at inducing activity-dependent plasticity” by the utilization of robotic trainers**, epidural electrical stimu- 
: lation?-!!, body-weight support trainers!”, brain-machine interfaces!)!*-5, transcranial magnetic stimulation!®!” 
: and surface functional electrical stimulation (sFES)'*!”. Recently, our group has shown that a training proto- 
col (the Walk Again NeuroRehabilitation protocol (WANR))!*”° — induced a significant level of neurological 
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recovery in a group of eight subjects with chronic complete paraplegia (seven AIS A, one AIS B), by combin- 
ing assisted walk training with robotic walkers, electroencephalography (EEG) -based brain-machine interface 
(BMI), and continuous visuotactile feedback. Extending our previous findings, the present study describes our 
group’s effort to further enhance clinical neurological recovery in severe cases of SCI by providing more selec- 
tive lower-limb musculoskeletal recruitment through sFES, while maintaining our fundamental philosophy of 
activating both ascending and descending neural pathways during the neurorehabilitation process. As such, the 
BENR (BML, sFES, NeuroRehabilitation) protocol described here integrates four key elements to potentiate recov- 
ery in patients with SCI: muscle activation through sFES”!-** (see for a review), balance control through body 
weight support”, real-time decoding of motor command (BMI)'*”° and sensory feedback through a portable 
haptic device”®. 

In recent years, BMI-FES (systems that use BMI to detect patients’ voluntary motor commands to trigger 
their muscle contractions through an FES) have been extensively studied as rehabilitation tools (some research- 
ers call it therapy or partial restoration) for patients with severe cases of stroke’”-”’, to potentiate recovery of 
upper-limb?****! or lower-limb motor function* (see** for a review). 

On the other hand, several studies with SCI patients have shown remarkable results when using BMI-sFES 
technology as an assistive device, by bypassing the spinal cord lesion and permitting the patient to perform motor 
functions***°. However, to date, little is known about the potential of the BMI-sFES approach as a true rehabil- 
itative tool/therapy for patients with SCI. To our knowledge, only one study with rats has rigorously tested this 
hypothesis: Bonizzato and colleagues’” recently showed that a training paradigm integrating BMI with epidural 
stimulation accelerated and enhanced the animal’s neurological recovery when compared to training that used 
only the epidural stimulation. 

Here, we studied both the assistive and the restorative aspects of a novel BMI-sFES system with two patients 
with chronic SCI. First, we demonstrated the validity of each aspect of our approach: a custom 16 channel sFES 
system, a closed-loop proportional- integral (PI) controller to cope with muscle fatigue, the integration of a port- 
able haptic device to cope with the lack of lower-limb sensory functions and a novel BMI protocol based on the 
detection of left and right leg motor imagery. Next, we tested our neurorehabilitation protocol with two patients 
with SCI (originally AIS A) who had been previously trained with the Walk Again Neurorehabilitation (WANR) 
protocol and, as a consequence, converted to AIS C”? (Supplementary Tables S1, $2). At the onset of the current 
protocol both patients presented motor functions more than three segments below the motor level (presence of 
knee extension (L3) for both patients); however, their overall motor score at that point in time was low (lower 
extremity motor score of 4 for P1 and 2 for P2 out of a maximum possible score of 50). To identify the effect of 
the BENR protocol, we ran a series of neurological*’, neurophysiological and functional (WISCI II)*’ assessments 
at the onset and at the end of training. Overall, we observed a significant improvement in muscle responses and 
reduction of muscle fatigue in our two patients in response to the sFES. Patients also experienced an increase in 
muscle volume and functional improvement for walking (measured as the ability to walk 10 meters in full load, 
using only passive assistive devices, such as walkers, orthoses, and crutches). More remarkably, we also docu- 
mented clinical neurological improvement in both patients. Indeed, one of these patients reached a substantial 
increase of 9 points for the lower extremity motor score (LEMS) after 22 sessions with the BFNR training. 


Results 

Two individuals with chronic paraplegia due to SCI, designated as P1 and P2 (40 and 32 years old; time since 
lesion: 4.5 and 10 years) with thoracic level injuries (patient P1 lesion at T7, and P2 at T4) were enrolled in our 
protocol (Supplementary Table $3). Before the sFES training, these patients had followed the WANR training for 
28 and 34 months!” respectively (Fig. 1A). Both patients had experienced sensory and motor improvements 
during the WANR and had been converted from AIS A to AIS C by the time of onset of the current protocol. 
First, we validated our setup for 6 months, for a total of 25 sessions, with patient P1. During this period, we tested 
our conditioning protocol (12 sessions), validated the closed-loop sFES stimulation (one session), validated the 
sFES-generated locomotion (10 sessions), and performed the integration of the tactile feedback with our training 
protocol (two sessions). Next, we tested the BFNR (BMI, sFES, NeuroRehabilitation) protocol with both patients. 
To isolate the effect of our training protocol, a 2-month washout period was given before the onset of the BFNR 
(Fig. 1A). The subjects did not receive any locomotion training, nor training in an upright position during this 
period. Clinical assessments, including the lower-limb circumference perimeter measurement”, evaluation of 
walking function with assistive devices’ and neurological status exam* were performed before and after the 
training protocol for both patients (measurements M4 to M5). We also compared both patients’ neurological 
improvement rate during the BFNR protocol with their improvement during the WANR protocol (measurements 
M1 to M2 for P1, M1 to M3 for P2), as well as their spontaneous improvement measured before they started the 
WANR protocol (measurements M0 to M1). For patient P1, we also controlled for whether our pilot validation 
tests induced any changes (measurements M2 to M3). 

The BFNR protocol was comprised of three basic phases: (1) lower-limb sFES conditioning (FC); (2) 
sFES-generated locomotion training (FL); and (3) brain-controlled sFES-generated locomotion (B + FL) 
(Fig. 1B). Patient P2 performed nine FC sessions, followed by four FL sessions, and then 10 B+ FL sessions. One 
extra FL session was recorded after the fifth B + FL session for analyses of patients’ walking aptitude using the 
sFES. Patient P1, who had already followed a conditioning phase during the pilot validation phase, had two FC 
sessions followed by the same sequence of FL and B + FL sessions as patient P2. 

During all BFNR phases we used a custom 16-channel sFES system” targeting the following muscles in both 
legs (Fig. 1C): gluteus maximus (GMx), gluteus medius (Gmd), rectus femoris proximal (RFP), hamstrings (Hs), 
vastus lateralis (VL), tibialis anterior (TA), gastrocnemius (Gs) and soleus (Sl). 

During the FL and B+ FL phases, patients were partially suspended (65-70%) with a body weight support 
system (Fig. 1D, Supplementary Fig. S1) (Zero-G, Aretech LLC., Ashburn, VA). During their walking over a 
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Figure 1. Methodology. (A) The timeline for experiment protocol, and six clinical measurements, ranged from 
M0 to M5. The training frequency (per month) is reported in each cell. WANR: Walk Again Neurorehabilitation 
Protocol'*°; sFES: surface functional electrical stimulation; BFNR: BMI and sFES neurorehabilitation. (B) 
Number of sessions per training phase for the BFNR protocol. (C) Eight lower-limb muscles are bilaterally 
stimulated with the sFES: three muscles for hip mobilization (flexion, extension, and abduction), two for the 

knee (flexion and extension) and three for the ankle (one dorsiflexor and two plantar flexors). (D) Patients were 
secured by a body-weight support system (Zero-G, Aretech LLC., Ashburn, VA) and used a walker for stability. 

A stimulation model, containing a reference kinematic profile and gait phases (detailed in subfigure G), was used 
as a base for the feedforward current stimulation. Patients’ hip and knee angles were measured in real-time. The 
command of the Proportional Integral (PI) controller was computed based on the error (ey) between the current 
joint angle (0,,) and the reference kinematics (0). We used a sigmoid function to convert the PI command into 
feedback currents for the flexors (Ig...) and extensors muscles (I.,,). The feedback and feedforward currents were 
summed to produce the actual values applied to the electrodes. (E) EEG electrodes were placed around the medial 
longitudinal fissure. Channel Fz was used as the reference and the Fpz as Gnd. Arm and leg sensorimotor areas are 
schematized on the figure on the right; Primary Motor (M1) and Sensory (S1) cortices, Pre-motor Cortex (PMC), 
Supplementary Motor Area (SMA) and posterior parietal cortex are shown. (F) The tactile shirt: six vibrators 
aligned on the subject's ulna bone. The subject felt a continuous tactile stimulation going from the wrist to the 
elbow by the swing phase of the ipsilateral leg and a simultaneous stimulation on all three vibrators at the onset of 
the stance. (G) sFES stimulation pattern for the lower-limb muscles to reproduce the eight sub-phases of a normal 
gait. The expected range of motion and the proposed stimulation current are shown for hip F/E, knee F/E, ankle 
dorsiflexion/plantar flexion and hip adduction/abduction. The pulse width and frequency were fixed. Three types 
of ramps were used to reproduce the gait cycle realistically. 


four-meter-long linear track, patients assisted themselves with a low-cost walker. The session was supervised by 
a physiotherapist or a physician. The patients’ hip and knee joint angles were measured every 7 ms. A PI control- 
ler adapted the stimulation amplitude in real-time to the muscle conditions determined by the lower-limb joint 
angles measurements. Two additional elements completed the setup: (a) a 16-channel EEG electrode (Fig. 1E) 
placed around the medial longitudinal fissure, densely clustered over the putative leg sensorimotor cortical area to 
detect the cortical activation for leg motor imagery; and (b) a portable haptic device”® that permitted the patients 
to perceive the position of the stimulated leg on their upper-arm (Fig. 1F). Continuous waves of tactile vibrations 
delivered to the patients’ forearms coincided with the stimulation of the lower-limb muscles to indicate the begin- 
ning and end of the swing phase of a stride. 

The preparation, followed by the training session, required approximatively 95 minutes, divided as follows: the 
sessions started with 15 minutes stretching, targeting both the upper-limbs, trunk, and lower-limb muscles. Next, 
approximately 35 minutes were used during the session preparation (placement of the EEG and sFES electrodes, 
placement of the position sensors and portable tactile feedback, placement of body weight support harness and 
moving patient to an upright position). The training lasted 45 minutes. We developed a novel sFES stimulation strat- 
egy that used time-based muscle contractions considering the eight sub-phases“ of a physiological gait pattern’) 
(Fig. 1G): (1) initial foot contact; (2) load response; (3) mid-stance; (4) terminal stance; (5) pre-swing; (6) initial 
swing; (7) mid-swing; and (8) terminal swing. To reproduce adequate gait during each sub-phase, we fine-tuned the 
stimulation time, the stimulation ramps up/down, the amplitude and the frequency for each muscle. 
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Figure 2. Setup validation. (A) Validation test of the PI controller; range of motion of the knee joint with and 
without the PI controller during a knee extension task with P1. We compare the joint angle produced by the 
controller in both cases (mean + SD, n= 56) to the expected target. (B) Root mean square error between the 
desired and measured knee angle (mean + SD) for both patients. Two-sided two-sampled t-test is reported. 

(C) Current applied without PI controller and with PI controller for the first and second half of the attempts 
(mean + SD, n= 56). (D) The amplitude of current for the vastus lateralis (VL) muscle during the conditioning 
sessions for patient P1. The duration of the session was set to 10 minutes and increased by 5 minutes every 
session (max. 40 min). When the therapist observed a clear sign of fatigue, the current was increased by five 
mA. The initial amplitude of a session was always set to 60 mA for this patient. (E) Mean + SEM of the time 
between two changes of current amplitude, considering all muscles, during the first 20 minutes of conditioning 
sessions three to nine for patient P1. Red line shows the linear regression of the means and corresponding linear 
function. (F) 3D reconstruction of the gait based on motion capture data measured during an FL session with 
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P2, and (G) angles for hip, knee ankle in the sagittal plane (mean + — SD over 54 full gait cycles). The lower part 
of the graph reports if the movement was correct (green) or not (orange or red). We report in orange the period 
where the movement was correct but the amplitude of the movement smaller (arrows pointing down) or higher 
(arrows up) than expected (small or big arrows describe if the difference was respectively in the 0-10° or 10-20° 
range). (H) Self-reported sets of comfortable walking speeds for patient P1. Walking speed varied on each run 
and the patient was asked to report if the speed was comfortable or not and if it was his favorite speed of the 
current session. (I) An experimented physiotherapist, blinded to the experimental conditions used an 11-item 
visual evaluation scale on qualitative and kinematics aspects of patients’ P1 and P2 sFES-generated walk when 
the tactile shirt was turned off (blue dot) or turned on (orange triangle). 


Pl controller validation. We introduced a PI controller to adapt the electrical stimulation pattern of the 
sFES system, based on muscle responses, to generate a reliable gait and cope with variance regarding muscle 
responsiveness and fatigue. To illustrate this effect, Fig. 2A compares the knee angle for patient P1 during a 
sFES-generated knee extension task in a seated position, when the PI controller was turned on (blue line) or off 
(grey line). To guarantee well-controlled conditions, only the knee joints were controlled for this test, while the 
other joints remained immobilized. In both conditions, the same desired target angle (black line) is used for com- 
parison. The test was repeated with both patients. We found that the root mean square error (RMS) for the PI-on 
condition (left knee 8.2°, right knee 7.8°, for P1, 11.2° and 9.3° for P2) was significantly smaller compared to PI-off 
(left knee 14.5°, right knee 14.4° for P1, 15.5° and 17.6° for P2) (two-sided two-sample t-test attests the significant 
difference (t(110) = 15.13, p=0 for P1; t(110) = 15.20, p=0 for P2) (Fig. 2B). These findings confirm that the PI 
controller produced more reproducible responses to the patients even in the presence of muscular fatigue. 

To cope with the increase in fatigue, the PI controller increased the amplitude of the stimulation for the last 
part of the sessions (Fig. 2C). The mean current employed was only 6% higher than in the PI-off condition for P1 
and 7.8% for P2. Considering the entire session, the mean currents for both conditions were comparable for both 
P1 (mean current of 49 mA for PI-on and 47 mA for PI-off), and P2 (42.51 mA PI-on, 39.96 PI-off), suggesting 
that the use of the PI did not notably increase the necessary current for stimulation. 


sFES-Conditioning validation. The FC phase was introduced to delay the onset of muscle fatigue while 
increasing muscle responses. Three factors were gradually adjusted during this phase: the stimulation time, the 
amplitude and the number of muscles stimulated simultaneously. The first training session was set to 10 minutes 
whereas the following sessions were gradually increased up to 40 minutes. At the onset of the sessions, the stimu- 
lation amplitude was set to the minimum current that produced a clear muscle contraction. Then, if the therapist 
observed a notable decrease in a given muscle response, the corresponding stimulation current was increased by 
5mA (maximum: 100mA). 

An example of stimulation amplitude is shown for nine sessions of VL muscle for patient P1 (Fig. 2D). For 
session three, during the first 20 minutes of training, it was necessary to increase the current five times (from 
60 mA to 80 mA) to observe a muscle activation that led to an overt movement. On the following day, at the same 
moment throughout the session, only 70 mA were needed to generate the same response. This positive trend 
continued throughout the training. Thus, after the sixth session of conditioning, the patients’ VL muscle response 
did not decay after 40 minutes of stimulation at 60 mA. 

We calculated the mean time before fatigue for all muscles during the first 20 minutes of the session. 
Considering only sessions that lasted 20 minutes or more (to avoid bias due to the session length), we observed a 
significant linear increase of the average time before fatigue (Fig. 2E, y= 0.62x + 11.03, P=0.01), confirming that 
the patients’ muscle responses improved following FC sessions. 


sFES stimulation strategy validation. Gait-analysis based on 3D construction of motion capture data of 
patient P2 revealed that the majority of the markers of the physiological walk were present (Fig. 2E, detailed anal- 
ysis in Supplementary Fig. $2, Supplementary Video S1). Indeed, on initial contact, we observed the accurate hip 
flexion, and the expected neutral position for the knee and the ankle. Through the mid-stance phase, we observed 
an accurate, neutral position for both the hip, knee and ankle joints. For the pre-swing, as expected, we observed 
the flexion of both the hip and the knee and the ankle plantar-flexion. We documented the flexion of the hip 
and the knee during the initial swing, and a neutral position of the ankle during the mid-swing. On the terminal 
swing, the hip flexion was correctly maintained, the ankle was in neutral position, and the knee extended. 

In three cases the range of motion was smaller than the normal gait: hip extension during the terminal-stance, 
the second peak of knee flexion during the swing phase, and the ankle dorsiflexion during mid and terminal 
stance. The hip extension and the ankle dorsiflexion limitations reduced the overall step length, whereas the lim- 
itation on the knee flexion reduced the foot clearance during the swing. Nevertheless, none of these limitations 
was severe enough to compromise the patient’s gait. 

We also wanted to be sure to use the fastest walking pace, to maximize the number of steps per session, while 
guaranteeing a comfortable and safe training environment for the patients. We performed a systematic evalua- 
tion of the maximum comfortable speed during six sessions with patient P1. For each run of 12 steps, we varied 
the speed and asked the patient whether the run was comfortable or not and whether the run was the patient’s 
favorite among the ones achieved in the current session. The patient was told before the session that the speed 
could change from one run to another, but no further information was given during the session. Over time, the 
patient reported feeling comfortable with speeds that were gradually increased (Fig. 2H); while the threshold for 
comfortable speed was between 6 and 8 seconds per step (s/step) during session seven, it reached a maximum of 
3 s/step on the 10" session. By then, the preferred speed improved from 8 s/step to 3 s/step. 
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Haptic feedback validation. SCI induces a loss of tactile and proprioceptive feedback information that 
is needed to generate a correct walking pattern***. To overcome this limitation, we integrated a portable haptic 
device, developed by our group (named the tactile shirt)”°, composed of arrays of vibrators to display relevant 
events using a tactile illusion called the apparent movement”. 

We ran two sessions with patient P1 to evaluate the difference of behavior in the presence or absence of the 
tactile feedback (Fig. 21). An expert physiotherapist, blinded to the conditions of the experiment used an 11-item 
visual assessment developed by our team to evaluate differences in terms of the patient’s adaptation to our loco- 
motion training setup (questions 1-5) and the patient’s gait kinematics during the sFES-generated locomotion 
(questions 6-11) (Supplementary Table S4 for details). The presence of the tactile shirt did not positively or nega- 
tively influence the patient’s trunk alignment, his need for compensatory movements, or his coordination with the 
walker. As expected, neither did we observe any substantial difference in any of the kinematic-related questions: 
the presence of the tactile feedback did not change the patient’s muscle responses during the stance (questions 
6-8), or the swing (questions 9-11) phases. 

On the other hand, we observed that the patient developed a tendency to look less at his own body during the 
walk when the tactile feedback was present (question 4). This phenomenon allowed patients to express a better 
understanding of the position of the leg in space, as well as the movement of the lower limbs. During the sessions, 
the therapist instructed the patient about correct trunk posture, upper and lower-limb participation, as well as the 
best way to coordinate the walker. We noticed a reduction in the need for these verbal instructions when tactile 
feedback was employed (question 5). Overall, we found that tactile feedback was useful for the patient to perform 
the task more independently. 


BFNR validation. After completing the pilot validation tests, we applied our BFNR protocol to both patients. 
Here, we present a quantitative and qualitative analysis of the patients’ walking with our sFES system, their per- 
formance with the BMI and their performances with the setup integrating the BMI and the sFES. Overall, both 
patients managed to walk correctly using the sFES and the stimulation paradigm proposed in this project (see 
Supplementary Movies S2 and S3 for runs with both patients). 

To have a complete view of the behavior of the two patients with our setup, we carried out a qualitative anal- 
ysis of the patients’ sFES-generated walking patterns during their last FL sessions. Figure 3A shows the hip and 
knee range of motion for these sessions and Figure 3B the detailed study carried out by a trained physiotherapist 
blinded to the experiment. 

Considering the metrics for high-level coordination, we did not find notable differences between the patients; 
both patients had good trunk alignment during the tasks; they, nevertheless, used compensatory movements to 
maintain their upper body. As patients used the tactile shirt in both sessions, we found results consistent with the 
pilot tests for P1. Basically, both patients did not need verbal instructions, nor did they need to constantly look 
at their bodies during the sFES-generated locomotion. During the stance phase, both patients had good knee 
position but poor plantigrade position during single support. Patient P1 had expected responses during the swing 
phase, with his foot rarely dragging on the floor. Patient P2 had difficulties in this aspect of the swing phase, due to 
a smaller response of the tibialis anterior muscle. We hypothesize that this lack of muscle response was due to an 
observed tension in this patient’s gastrocnemius (Gs) muscle (which is the antagonistic muscle of this movement). 
Likely, this resulted from his positioning in his wheelchair since his knees and feet usually stayed flexed in his 
wheelchair, which could lead to increased tension in the Gs muscle. 

Both patients in this protocol had been previously trained with the BMI protocol using an arm or leg motor 
imagery as described in the WANR protocol'*”® for over 2 years. However, the specific BMI protocol proposed 
in the current study was new to both patients. Each B+ FL session started with an 8-minute open-loop training 
block where patients were randomly instructed to imagine moving their left or right leg. The recorded EEG data 
was used to train the EEG classifier. The classification accuracy for both patients was above chance (73.9% + 3.6% 
for P1 and 83.2% + 8.2% for P2) (Fig. 3C). Patient P2’s classifier accuracy was significantly higher than for P1 
(t-test, P= 0.002). To confirm that patients effectively used leg motor imagery during the B + FL sessions, we 
analyzed the principal components calculated with the common spatial pattern filter (CSP)*° (Fig. 3D). The CSP 
algorithm detects the EEG electrodes that maximize the variance for the two classes (here, left leg versus right leg 
motor imagery). For left leg motor imagery, we found the main activation over the C2 electrode for both patients 
(see Fig. 1E for electrode placement), while for right leg motor imagery, the CSP filter detected the highest vari- 
ance over electrodes CCP 1h, confirming cortical activation in the leg motor area. 

Next, patients learned to perform leg motor imagery to trigger the stimulation of the corresponding leg 
through an EEG-based brain-machine interface (Fig. 3E). An array of LEDs informed the patients of the cur- 
rent state of the trial and the brain activity classification. A trial was defined as follows: after a relax period of 
4 seconds where the patients could adjust themselves, and 1-second focus time, the patients had a 4-second time 
window to perform the motor imagery (Fig. 3F). If the patient managed to maintain the correct motor imagery 
for 2 seconds, the trial was considered as successful, and the corresponding step was triggered. If the timeout 
was reached, the trial was registered as a failure, and an automatic step started. The time to trigger a step, there- 
fore, ranged between 2 and 4seconds. During each session, patients performed 72 steps divided into six runs 
separated by a 1-minute break. 

The number of correct steps per run is reported in Fig. 3G. Patient P2 outperformed patient P1 regarding the 
number of successful trials per session by performing 88% of the runs above chance and more than one-fifth of 
the runs with a perfect score (Fig. 3G). Throughout the training, we observed a clear improvement in terms of 
control accuracy for patient P2. Indeed, when considering the second half of the training, 25 out of 30 runs were 
significantly better than chance (mean + 2*SD). 
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Figure 3. Protocol validation. (A) The ranges of motion for hip and knee joints for both patients. The eight 
gait subphases are shown inside the gray boxes. Dashed line shows the onset of the swing phase. (B) Qualitative 
and kinematics’ aspects of patients’ sFES-generated walk. (C) Patients’ classifier accuracy for the EEG training 
phase (mean + SD, on the 10 B + FL sessions and ttest). (D) The coefficient of the principal component found 
by the CSP algorithm for a B + FL training session for left and right leg motor imagery. Electrode placement 

is the same as in Fig. 1E (three electrode names C3, C4 and Pz are shown as references). (E) For each step, the 
following sequence was reproduced: the patient relaxed and adjusted himself for 4seconds, focused during 

1 second and had 4 seconds to produce the motor command. During this window, ifthe patient maintained 
the correct motor imagery for 2 seconds, the step was triggered and considered as successful. Otherwise, if the 
timeout was reached, an automatic step was triggered, and the trial was considered as failed. (F) Example of 
successful and failed step. RMI/LMI: right/left motor imagery, NS: no state, t,,4: cumulated time of left motor 
imagery (G) Number of correctly performed steps for all patients for all B + FL runs over 12 steps. The chance 
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level is reported with black dashed lines, 95% interval of confidence is shown in gray. (H) Mean + SE time to 
control a step with the B + FL protocol. Each dot represents 12 steps ofa run. Fastest possible time to perform 
the task was 2 seconds, and the chance level is shown with a black dashed line, 95% interval of confidence in 
gray (3.23-3.37 [s]). Linear regression is represented by dashed lines for each patient. 


Patient P1 had just 50% of the runs above chance level. This shows that the BMI protocol and the need to 
maintain the leg motor imagery for 2 seconds was particularly difficult for patient P1 who, despite having a good 
classifier score (on average >70%), did not manage to perform the BMI control above chance level. 

We also analyzed the mean time to perform a step for each session (Fig. 3H). We found a small but statisti- 
cally significant trend towards reducing the time to perform the task in patient P2 (y= —0.11x+3.44, p=0) and 
patient P1 (y= —0.03x + 3.7, p = 0.03) with training. Altogether, we observed significant control performances in 
patient P2. Patient P1, who started with lower performance rates, improved (although at a slower rate) with the 
training and could have potentially reached better control with more extended BMI training. 


Clinical improvements. Following the BFNR protocol, we observed notable physical improvements in both 
patients. This was first verified by an increase in muscle volume through the perimeter assessment**. We found, as 
expected, an increase of the thigh volume of 276 cm} in patient P1 and the lower-leg volume of 636 cm? (Table 1, 
Supplementary Fig. S3). Patient P2 also exhibited this effect, with an increase of 838 cm? of thigh volume and 
280 cm? for the lower-leg volume. 

We next analyzed the patients’ walking functions using assistive devices. For this, we employed the WISCI II 
evaluation’’, which assesses the amount of physical assistance needed, as well as the required devices (orthosis 
and gait assistive devices), for a patient with SCI to walk 10 meters (Table 2). The exam was performed with no 
body weight support and no sFES. At the onset of the BFNR (M5), patient P1 was able to walk 10 meters using a 
lower-limb orthosis (HKAFO: hip-knee-ankle-foot orthosis), a triangular walker and required additional physi- 
cal assistance for hip and trunk stabilization (WISCI score of 6). Following the BENR protocol (M6) this patient’s 
functional score improved to 9, as he required no physical assistance to perform the task. He also managed to 
perform the same test with crutches and one assistant (score of 7). Additionally, P1 was able to accomplish the 
task 19 seconds faster than at the onset of the BFNR. No notable changes were observed during the pilot valida- 
tion tests (M3 to M4). Considering the necessary level of assistance to perform the 10m task, we did not register 
changes for patient P2 (score of 6 at the onset and the end of the protocol), he nevertheless managed to perform 
the test 14 seconds faster. 

Both patients experienced a small decrease in the resting heart rate (HR) throughout the protocol (81 bpm to 
74 for P1,79 bpm to 76 for P2, Table 2). Similarly, the HR increase due to the effort was smaller at the end of train- 
ing for P2 (HR increase of 41 bpm at the onset of the training (M5 and 23 bpm at the M6). Patient P1 improved 
during the pilot tests (an increase of HR of 30 bpm was observed at M2, and an increase of 1 at M3). During the 
measurement following the BFNR, this patient managed to perform the task with less assistance which translated 
into a higher increase in HR (36 bpm). 

The Borg fatigue assessment is a subjective self-reported level of exertion during exercise ranging between 
0 (no fatigue) to 10. Patients were asked to report the level of fatigue in their upper-limbs. We observed a small 
decrease in the fatigue assessment following the BFNR for patient P1. Blood pressure and oxygen saturation were 
also measured before and after each WISCI assessment (Table 2). All recorded values were in the normal range. 

No clear neurological improvement was observed in the patients’ sensory domain (light touch, pin-prick sen- 
sation) during the BFNR (Fig. 4A). Patient P2 had an improvement rate comparable to the one he had during the 
first 28 months of WANR, while patient P1 stagnated. 

On the other hand, we documented significant motor improvement in both patients. The improvement was 
assessed through the lower extremity motor score measurement (LEMS) of five lower-limb muscle groups (rec- 
tus femoris proximal and distal, tibialis anterior, extensor hallucis longus and gastrocnemius). For a complete 
view, we recorded seven additional lower-limb muscles (hip adductors, gluteus maximus, gluteus medius, medial 
and lateral hamstring,flexor hallucis longus and extensor digitorum longus). The scoring method followed the 
standard ASIA motor exam conventions, ranging from 0 (no contraction) to 5 (normal contraction). We show 
the LEMS score for both patients in the lower panel of Fig. 4A, and the details per muscle in Table 3. For patient 
P1, we observed, a 1 point LEMS increase during the sFES pilot phase and a gain of 3 points during the BFNR 
phase. This was an encouraging result, as it demonstrated that even at the chronic phase of an SCL, it is still pos- 
sible to observe neurological improvement. Notably, patient P2 exhibited a much higher motor improvement 
(Supplementary Movie S4); starting with an LEMS of 2 measured at the onset of the BENR training. This patient 
reached a final score of 11 points. Considering all 12 muscles, this patient went from a score of 6 to 21 (Table 3). 
By the end of the training, patient P2 could contract the Rectus Femoris Distal (RFD) muscles with a score of 2. 
This was confirmed both clinically and thorough EMG recordings (Fig. 4B) showing the presence of significant 
contractions at the end of the BFRN. 

We next analyzed the proportion of muscles that improved with the training, considering the muscles that 
were stimulated with the sFES separately from those that were not (lower part of Table 3). For patient P1, no clear 
improvement was observed during the pilot tests. During the BFRN for this patient, none of the non-stimulated 
muscles improved, but 5 out of the 16 stimulated muscles improved. For patient P2, we observed an improve- 
ment in one out of the eight non-stimulated muscles and 14 out the 16 stimulated ones. We observed that the 
best motor improvement occurred in muscles that were stimulated with sFES, with notably better responses for 
patient P2. 


SCIENTIFIC REPORTS | (2019) 9:6782 | https://doi.org/10.1038/s41598-019-43041-9 8 


www.nature.com/scientificreports/ 


—0.7 2.1 
1.0 14 
Hip 276 ai 838 
0.6 3.5 
0.1 3.7 
0.2 2.05 
Knee 2.5 0 
0.8 1.6 
0.4 1.8 
2.5 2 
3.9 636 1.7 280 
43 1.15 
2.5 0 
Ankle 3.1 —2.75 


Table 1. Difference between the post-BFNR (M5) and Pre BFRN (M4) measurements for the lower-limb 


perimeter. 
WISCI 
Score 6 6 6 9 6 6 
Time to walk 10m [s] | 99 98 108 89 66 52 
Vital signs (pre) 
PA (mmHg) 130 x 90 120 x 80 130 x 90 120 x 80 120 x 70 120 x 80 
Heart rate (bpm) 94 92 81 74 79 76 
Sat O, (%) 99% 97% 96% 97% 96% 96% 
Borg? (Upper limbs)’ | 0 0 0 0 0 0 
Borg (Respiratory)* 0 0 0 0 0 0 
Vital signs (post) 
PA (mmHg) 130 x 85 120 x 80 130 x 90 130 x 80 130 x 80 130 x 70 
HR (bpm) 124 93 96 110 120 99 
Sat O, (%) 98% 92% 97% 99% 99% 98% 
Borg (Upper limbs) 3 4 5 3 2 2 
Borg (Respiratory) 2 2 5 i] 2 1 


Table 2. Walking functions assessment with assistive devices with the Walking Index for SCI (WISCI II) 
evaluation’. We evaluate the time to perform the 10 m task and vital signs recorded before and after the test. 
*The Borg fatigue assessment is a subjective self-reported level of exertion during exercise; *Level of arms 
fatigue; ‘Measurement the general physical fatigue (heat, sweat or dyspnea). 


Overall, the motor improvement in both our patients could not be compared to the type of spontaneous 
improvement rates reported in the literature (scores <1 point when the training starts 26 weeks after the injury”) 
or those recorded for our patients prior to the start of training in our laboratory (0 for both patients, M1-M0). 
For patient P2, the improvement observed following the BFNR (9 points after just 5 months) was even higher 
than the one he experienced during the WANR protocol (i.e., a 4 point improvement during the first 28 months). 


Discussion 
This paper introduces a novel, non-invasive neurorehabilitation protocol for locomotion training for patients 
with severe chronic paraplegia resulting from spinal cord injuries, which targets both musculoskeletal training 
and corticospinal plasticity. Our setup integrated several technical novelties: a new sFES paradigm targeting mus- 
cle activation to produce locomotion, a BMI paradigm based on detection of left and right leg motor imagery to 
trigger the sFES, and sensory substitution to provide patients with tactile feedback from the lower limbs. We also 
introduced a step-by-step conditioning/training protocol and validated it with two individuals with paraplegia. 
Using this novel approach, we successfully tested a stimulation protocol to reproduce a smooth gait pattern; 
patients were able to use our setup to walk with partial suspension and little or no external help. This result 
was possible thanks to muscle conditioning preceding the sFES-generated walk. Following an SCI, muscles 
with myotomes originating below the lesion level undergo severe disuse atrophy*”“*. Several years after the 
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A. Neurological evaluation B. EMG signals for Rectus Femoris Distal 
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Figure 4. Neurological improvements. (A) Six neurological evaluations (M0 to M5) for both patients. 
Sensory score improvement is calculated using standard ISNCSCI assessment, compared to the score at 

the onset of the training (for both patients, the score at time = 0 was subtracted from all evaluations). The 
lower extremity motor score (LEMS) is based on clinical evaluation of five key lower-limb muscles (0 for no 
contraction, 5 for normal), the maximum score is 50. (B) Rectus muscle EMGs for patient P2 before and after 
the BENR protocol. Signal was filtered between 5 Hz and 250 Hz with a Butterworth 4"-order band-pass, and 
a 60 Hz notch filter. The envelope was obtained after rectification of the filtered signal and a 1 Hz Butterworth 
high-pass filter. Thick black lines represent the physiotherapist’s instruction time to contract the muscles. The 
black horizontal line is the mean + 3 x SD of the activation as calculated during the baseline periods. 


lesion, the volume of muscle fibers is reduced, becoming more and more prone to the process of fat infiltration. 
Concurrently, slow fatigable fibers tend to transform into fast fatigable fibers*’”. As a result of our training proto- 
col, our patients experienced an increase in the volume of their leg muscle fibers and a parallel increase in their 
resistance to muscle fatigue”"?->). 
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Hip flexor Mlopsoas rectus femoris 1 jaja Ja fa fala Ja fo fola Ja 
Knee extensor Vastus, rectus femoris distal 1 1/1 1/1 1/1 241 1 |2 2 
Keymubels Ankle dorsiflexor Tibialis anterior 0 jojo jojo Jojo jojo joji fa 
Hallux extensor Extensor hallucis longus 0 0 {0 0 {0 0 {0 0 {0 0/0 1 
Ankle plantar flexor Gastrocnemius, soleus 0 041 010 0 }1 1/0 0}1 1 
Hip adductor Adductors 2 2 /|2 2/1 2/1 £ Nd 1d 1 
Knee flexor Medial hamstring 0 0 {0 0 {0 0/0 0/0 ame | 1 
Knee flexor Lateral hamstring 0 0 {0 1 |0 1 |0 Ee O7;1 1 
Auxiliary muscles Hip abductor Gluteus medius il 1 jl ae 1 jl 1 |0 ae 1. 

Hip extensor Gluteus maximus 2 241 Lo Ty. 2)1 T-| a 1 
Toes extensor/flexor aoe digitorum 0 0 10 0 10 0 10 0 {0 0 {0 0 
Hallux flexor Flexor hallucis longus 0 0 {0 0 {0 0 {0 0 {0 0 {0 0 

LEMS (sum key muscles) 4 5 4 7 2 11 

Sum all muscles 14 14 12 16 6 21 

Regression | 2 0 0 
tien anne Stagnated | 14/16 11/16 2/16 
Improved | 2/16 5/16 14/16 

Non-sFES Regression | 0 1/8 0 

muscles (8 Stagnated | 8/8 7/18 718 

nnigclen) Improved | 0 0 1/8 


Table 3. Muscle score for five key muscles* and seven auxiliary muscles. The test is done without any sFES 
stimulation. The muscles that were targeted during the BFNR training are in underlined. The lower part of the 
table shows the number of muscles among the ones that were targeted (sFES muscles) and the ones that were 
not targeted with the sFES (Non-sFES muscles) that have regressed, stagnated or improved during the pilot test 
and the BFRN. 


Another important aspect of our strategy was the stimulation of a large number of muscles (16 lower-limb 
muscles) with precise timing. This was possible thanks to a custom-built programmable sFES stimulator” to 
target each muscle individually. The conventional sFES technique to reproduce locomotion uses stimulation 
of the peroneal nerve to elicit the triple-flexion reflex during the swing and stimulation of the gluteus and the 
quadriceps during the stance phase of the gait**°?**. This technique is limited by the high variability of the muscle 
response and the rapid habituation to the stimulation™. Instead, in our protocol, we decided to directly stimulate 
the lower-limb muscles to produce a consistent gait pattern. A major technical difficulty of our approach came 
from the inherent non-linearity of muscle responses™ (sigmoidal shape response type) and the effect of muscular 
fatigue due to the recruitment of muscles fibers in a non-selective, spatially fixed and temporally synchronous 
pattern®*°®. In SCI subjects, following the lesion, there is also a trend towards the substitution of muscle fibers 
into rapidly fatigable fibers*’. To cope with these difficulties, we proposed and successfully tested a closed-loop 
sFES controller using the joint angles of the lower-limbs to adapt the stimulation in real time. We also introduced 
online tactile feedback to help patients perceive gait events. Previously, our group has reported that this sort of 
feedback promotes cortical plasticity*® and provides patients with the type of lower-limb sensory information 
needed to improve their gait. Here, we observed that the use of tactile feedback increased the patients’ confidence 
and independence during the walk. Overall, using our setup and following our protocol, our patients acquired 
the ability to produce a smooth sFES-generated gait pattern. Indeed, for both patients, the main clinical features 
of human gait were respected: absence of crouching at stance, neutral position of the ankle at initial contact and 
during stance phase, as well as sufficient foot clearance during the swing. 

Our BFNR protocol was specifically designed considering the use of BMI for neurorehabilitation purposes 
(rather than assistance)*°*” for patients with SCI with minimal or no motor function. Therefore, the focus was not 
placed at achieving the highest level of control accuracy or speed, but rather to create a condition in which both 
afferent and efferent signals, converging at the level of the lesion, are engaged simultaneously. Thus, to trigger 
a step, patients had to perform and maintain leg motor imagery of the corresponding leg for two seconds. This 
protocol was particularly complicated for one patient who experienced more difficulties in control performance. 
Future work with a larger cohort of patients is necessary to address this issue; for example, a shorter trial-time 
could potentially facilitate the task. 

As the final outcome of this study, we observed that patients exposed to our protocol exhibited important 
clinical improvements, including, as expected, increases in muscle volume’. Moreover, we observed better out- 
comes in walking functions using assistive devices. As a result, our patients’ step frequency increased, and one 
patient was able to walk using less assistance at the end of the protocol. This was probably a consequence of better 


SCIENTIFIC REPORTS | (2019) 9:6782 | https://doi.org/10.1038/s41598-019-43041-9 


a a 


www.nature.com/scientificreports/ 


overall physical condition, which was indirectly inferred by the decrease in resting heart rate, smaller heart rate 
increase after the effort, improved muscular resistance, better coordination with walking devices due to the use of 
a walker and body weight support in the protocol, and the occurrence of a significant partial neurological recov- 
ery in the motor domain. Considering this latter component, partial motor neurological recovery was observed 
in different degrees: it was moderate in patient P1 and substantial in patient P2. Importantly, P2 was the patient 
who had good BMI control and good sFES responses. Further clinical trials are necessary to disambiguate if 
achieving good BMI-control accuracy is a necessary condition or even a determinant for patients to benefit from 
this training. 

Yet, there is one major question that remains open: what is the mechanism that induces this improvement? We 
hypothesize that, like in the case of our previous results with the WANR, the concurrent occurrence of cortical 
plasticity, induced by BMI use'!'*"" and spinal plasticity, promoted by sFES through ascending signals°*-*! was an 
essential agent for the observed neurological motor improvement in our study. Recent work with human subjects 
showed that the use of invasive epidural stimulation has the potential to induce partial neurological restoration 
in chronic complete SCI patients’. Importantly, a study with rats’ showed that the motor recovery was faster and 
more important when the epidural stimulation was paired with brain-machine interfaces compared to conditions 
where epidural stimulation was used alone. These findings provide clear support for our hypothesis. 

For both patients, the cortical activation during leg motor imagery was found close to the cortical leg 
sensory-motor area. The presence of activation in posterior areas was previously reported in patients with chronic 
complete SCI due to cortical reorganization towards the primary somatosensory cortex°*. 

Given the improvements obtained in this pilot study in terms of muscular volume and performance, better 
walking functions and neurological recovery, we propose that our BFNR protocol has the potential to become a 
valuable neurorehabilitation therapy for patients with SCI in the future. 


Methods 

Patients inclusion/exclusion criteria. Subjects were adults diagnosed with paraplegia, with traumatic SCI 
thoracic injury at the chronic phase of the lesion, emotionally stable and with the absence of offset comorbidities. 
The inclusion criteria for the present study considered patients that had previously participated in the WANR 
protocol and who exhibited evidence of upper motor neuron injury, clinically manifested by the presence of 
spasticity and positive response to spinal reflexes. The screening was performed to test their muscles’ responsivity 
to sFES before inclusion. Exclusion criteria included lower motor neuron injury, the absence of muscle response 
during sFES responsivity screening, a degree of spasticity exceeding a score of 2 (Ashworth scale), and a degree of 
osteoporosis (T-score) greater than —4. Two patients were included in this study. Prior to the intervention, patient 
P1 and P2 followed a neurorehabilitation protocol introduced by our team (Walk Again Neurorehabilitation 
Protocol, WANR)” which integrated the use of body weight support (BWS, Zero-G, Aretech LLC., Ashburn, VA), 
Lokomat, brain-machine interface (both arms and legs motor imagery), and tactile feedback”® for 28 months and 
34 months respectively. The two patients, referred to here as P1 and P2, were previously referred to as P5 and P6 
by Donati et al." and Shokur et al.?°”®, 


Study approval. The study was approved by both the local ethics committee (Associacao de Assisténcia a 
Crianca Deficiente, Sao Paulo, Sao Paulo, Brazil #364.027) and the Brazilian federal government ethics committee 
(CONEP, CAAE: 13165913.1.0000.0085). The experiments were carried out as a feasibility study that did not meet 
the clinicaltrials.gov definition of an Applicable Clinical Trial. All research activities were carried out in accord- 
ance with the guidelines and regulations of the Associacao de Assisténcia a Crianga Deficiente and CONEP. The 
participants signed a written informed consent before enrolling in the study, and also signed a written informed 
consent for open access publication (print and digital) of their images. The experiments were carried out at the 
Associa¢ao Alberto Santos Dumont para Apoio a Pesquisa (AASDAP), Sao Paulo. 


sFES conditioning. An initial screening, which tested all muscles individually, helped set the stimulation 
amplitude and frequency parameters for the conditioning. The pulse width was fixed at 300 1s for the whole 
program®. The conditioning phase aimed at preparing the muscles progressively to produce a muscle contrac- 
tion that allowed lower-limb movements of flexion, extension, and abduction, with the final goal to produce a 
sFES-generated gait. The stimulation time for each muscle increased gradually, up to 40 minutes maximum. The 
ramp up/down stimulation time decreased from one session to the next one. During the first eight sessions, the 
patient was maintained in the supine position, and the legs were positioned on rollers to allow free movement. 
The muscles were first stimulated individually, then in pairs and finally alternating between flexors and extensors 
(leg-by-leg pattern). The leg-by-leg pattern was applied in two phases: extensors of one leg were stimulated, while 
the flexors of the opposite leg contracted for 10-15 seconds. The muscles then rested for 2 seconds before the 
opposite pattern was applied. During the session, a physiatrist supervising the experiment evaluated the muscle 
response (visual evaluation of the range of motion of the joints and presence or absence of fatigue), the tonus 
behavior, the sensory perception of the stimulation and the ability to voluntarily improve the muscle contraction 
together with the stimulation. 

For the last session, the patient was supported in an upright position in a body weight support system 
(Zero-G, Aretech LLC., Ashburn, VA). The stimulation sequence, called stationary gait, differed from the 
leg-by-leg sequence due to the continuity of the stimulation to extend both lower-limbs. The following criteria 
were expected for a patient to continue with the FC phase in standing position: adequate muscle responses to 
allow the hip and knee extension, stable blood pressure, no signs of fatigue during the first 25 minutes, and no 
spasticity disturbing the stimulation pattern. 
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Body-weight support system. A robotic body weight support system (Zero-G, Aretech LLC., Ashburn, VA) was 
used to support 65-70% of the weight of the patient during gait. This percentage was selected to provide the min- 
imum suspension required to avoid a crouching position of the lower-limbs, ensuring good contact between the 
foot and the ground and reducing muscle effort to help avoid premature fatigue. 


Real-time tracking system. During all experiments, hip and knee joint angles were measured in real-time at 
148 Hz. The system used four inertial measurement units (IMU, Trigno TM Inertial Measurement Sensors, 
Delsys, USA). One sensor was placed on each femur (thigh) and tibia (lower leg) of the patient. The joint angle 
was computed based on the difference in orientation of the sensors on the adjacent segments”. The patient was 
asked to keep the trunk position straight throughout the training. At the first session, the sensors were positioned 
on the body relative to anatomical references (patella, tibial tuberosity, and malleolus) and we calculated the 
parameters needed for the algorithm. 


Description of the gait pattern and rationale. To produce the gait movement, we designed an activation sequence 
based on 16 lower-limb muscles stimulated non-invasively. The stimulation pattern, shown in Fig. 1G was deter- 
mined based on eight gait sub-phases, namely (1) initial contact, (2) load response, (3) mid-stance, (4) terminal 
stance, (5) pre-swing, (6) initial swing, (7) mid-swing and (8) terminal swing. The stride duration could vary from 
4 to 20 seconds. The highest joint range of motion of the human gait takes place in the sagittal plane where flexion 
and extension of the hip, knee, and ankle occur. We selected extensors and flexors for each joint. 

We describe here each muscle activation of the stimulation pattern, considering the start of the stance as the 
start of the gait cycle. In this position, during initial contact, the vertical reaction force is acting in front of the 
hip joint promoting an internal flexor moment that will induce hip flexion. The GMx is therefore activated to 
challenge this internal moment to extend the hip and to align the trunk and pelvis in the orthostatic posture. 
During the single support phase (that includes mid-stance and terminal stance sub-phases), the patients’ lack of 
lateral pelvic stabilization could potentially compromise hip joint integrity. To avoid this issue, the Gmd muscle 
was stimulated to stabilize the pelvis segment in the frontal plane. The Sl muscle was recruited to control the 
advancement of the tibia in mid-stance, together with the knee extensor, VL, to maintain dynamic stability and 
extension of the knee at stance. 

The pre-swing phase started when the opposite foot touched the ground. At this instant, the Gs was stimulated 
to assist the knee flexion, and the Hs stimulation reinforced this movement. This condition promoted lifting the 
heel off the ground. After this, during the initial swing, the RFP was stimulated to flex the hip to lift the foot from 
the ground. Then the RFP acted to counteract the external lower-limb extension moment promoted by gravity. 
The RFP stimulation during terminal swing also allowed better control of the downward movement of the limb 
towards the ground, preparing the limb for the next cycle (next initial contact). 

The Hs stimulation was progressively replaced by the VL to extend the knee gradually in the mid- and termi- 
nal swing. The TA stimulation promoted ankle dorsiflexion to keep it in a neutral position during the swing phase 
and to avoid dragging the foot on the ground. The TA stimulation decreased progressively, slightly before the foot 
touched the ground, to let the foot set down gently on the floor in a flat position, minimizing joint injuries. Three 
types of different speeds ramps were used (slow, middle, rapid). 


sFES Stimulator. We used a custom 20-channel stimulator for this project®*”*”. The stimulator sends 
charge-balanced, rectangular, biphasic pulses. Surface electrodes were used to stimulate 16 selected muscles as 
shown in Fig. 1C. Two sizes of electrodes were used: 5 x 10 cm (for GMx, Gmd, VL, Hs, Gs) and 5 x 5cm to target 
important muscles more locally (RFP) or smaller muscles (TA, Sl). The stimulator allowed the modulation of 
pulse width, frequency and current intensity at a refresh rate of 10 ms. The pulse width was maintained at 300 ps 
as this was previously found as an optimal value®. By default, a frequency of 30 Hz was used“. This electrostimu- 
lator and the risk analysis associated report has been approved by the SwissMedic Ethical Committee for Clinical 
Investigations (SwissMedic Ethical approval, Ref MD-2007-MD-0031). 


PI controller. The PI controller ran at 50 Hz and adapted the stimulation amplitude to reach the predefined tar- 
get angles (see Description of the gait pattern and rationale). For safety, the maximum current modulation was 
saturated within a range of 20% above or below the command. The parameters for the PI controllers were found 
using the pure PI control (i-e., without pre-established stimulation pattern) on healthy subjects to achieve typical 
joint trajectories (ramp target, sinusoidal target and kinematics curves from a normal gait cycle). Proportional 
and integral gain, Kp and Ki were respectively set to 0.5 and 0.001. 


PI controller characterization experiment. To evaluate the effect of the PI controller, we designed a specific 
experiment that isolated only the knee extension. The patient was sitting in his wheelchair with his feet hang- 
ing. The vastus of each leg was stimulated alternately to induce left and right knee extension. We designed the 
desired angle trajectory to be composed of ramps and plateaus to study the characteristics of the controller 
with constant and changing targets. We pseudo-randomized the trials where the PI controller was on (PI-on) 
and trials where it was off (PI-off) (three PI-off and three PI-on were shuffled every six repetitions). Only the 
vastus muscle was stimulated to act as knee extensor, and the knee flexion was performed passively by gravity. 
The whole experiment was composed of 100 repetitions (50 open-loop, PI-off, and 50 closed-loop, PI-on). The 
current amplitude model for the open-loop was composed of ramps and plateaus between two amplitudes. 
Those two amplitudes were selected during preliminary tests on the same day to obtain two levels of extension 
close to the desired target. 
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Custom visualization system and LEDs. During the B+ FL phase, two arrays of four LEDs were placed on each 
side of the walker to indicate the output of the EEG classifier and the experiment’s protocol. The left and rightmost 
LEDs informed the patients of the experiment with a simple color convention: black (turned off) for idle time, 
pink for focus, and left or right blue LED for corresponding leg motor imagery. At the end of each trial, the patient 
could see if he succeeded or failed the trial (green or red LED). Patients’ EEG classifier value was discretized in the 
[—1.5; +1.5] range and shown using the six remaining LEDs. 


EEG and BMI. We used the opensource software OpenViBE 0.16.0, for the acquisition and processing of EEG sig- 
nals®. EEG signals were acquired at 2000 Hz with the V-amp amplifier (Brain Products, GmbH). Sixteen channels were 
recorded and clustered around the leg representation of the primary sensorimotor cortex (Fig. 1E). We used actiCAP 
active electrodes, connected to the wireless MOVE (both Brain Products, GmbH), carried with the patient. The wireless 
receiver was directly connected to the amplifier via USB to a computer. EEG signals were band-pass filtered at 8-30 Hz 
(mu and beta), corresponding to the well-documented frequency band for motor-related EEG activity”. 

We then used two algorithms for the detection of motor imagery: the Common Spatial Patterns (CSP) algo- 
rithm“ as a spatial filter and Linear Discriminant Analysis (LDA) for classification. The CSP aims at learning 
spatial filters, such that the variance of the signals is maximized for one class (e.g., one mental imagery task) and 
minimized for the other class. The Linear Discriminant Analysis (LDA) algorithm was used to estimate a linear 
hyperplane to separate feature vectors from two classes)’!. In our case, if the LDA output was positive, the feature 
vector was assigned to the right motor imagery class. Otherwise, it was assigned to the left leg motor imagery 
class. Every BMI session started with 8 minutes of training (40 trials, 20 left and right leg motor imageries, ran- 
domized order). Each trial consisted of 1-second preparation, followed by 1.25s of cue presentation and a 3.75s 
motor imagery period. The intertrial time was randomly chosen between 1.5 and 3.5 seconds. 

Patients were instructed to produce motor imagery involving only one leg at the time as, e.g., ‘imagine kicking 
a ball with your right leg, ‘imagine making a circular movement with your left ankle; and to avoid motor image- 
ries that involved the movement of both legs such as ‘cycling’ or ‘walking? We used a 5-fold, cross-validation for 
calculation of the classifier accuracy. 

During this acquisition, the patient already had the sFES electrodes placed and connected to the stimulator 
(with the stimulator turned off); he was standing, in the BWS system, but without suspension and was supported 
in an upright position. Before the first BMI run, the patient remained for 30 seconds without motor imagery in 
a standing position to record the baseline classifier. The mean of this signal was subtracted from the classifier for 
the run to avoid any offset. The baseline could be calculated one more time before the fourth run if needed to 
compensate for any new offset. 

During the online control, we used the parameters for the CSP and the LDA that were calculated during the 
training phase. The only difference with the training phase was a threshold of 0.1 on the classifier value to reduce 
chances of false positives: we considered detection of left or right motor imagery only when the classifier was, 
respectively, smaller than —0.1 or above 0.1. 


Tactile shirt. The tactile shirt was composed of the three vibrators aligned on the ulna as described in**. Patients 
received continuous tactile feedback going from the wrist to the elbow (apparent movement as described in‘*) 
coinciding with the stimulation of the lower-limb muscles to indicate the beginning the swing (start of phase six, 
Fig. 1G). At the end of phase eight, they received stimulation of all three vibrators together for 600 ms. 


Calculation of the chance level for the BMI. ‘The baseline data was gathered during resting time with the same patients, 
collected at the onset of each BMI session. All experimental conditions were the same as those for the current experi- 
mental paradigm. Patients were using the same EEG cap, and setup (standing position). EEG signals were recorded for 
30 seconds prior to the session. We gathered all the baseline datasets and cut them into 4 time windows corresponding 
to step trials. Then we applied the exact same algorithm and simulated a sequence of runs based on these datasets. We 
report the statistics for chance level, obtained by simulating 600 steps (50 runs of 12 steps): 5.72 + — 1.82 (mean + —SD) 
correct steps performed over 12 steps run, and the time to perform a step is on average 3.31 + —0.84[s]. 


Session interruption and data exclusion. A clinical check was done before and during the sessions. If the patient 
presented abnormal low blood pressure or severe cases of spasticity before or during the session, the session was, 
respectively, not initiated or interrupted. 


Qualitative visual gait evaluation. Each evaluation session had six FL runs; each run had 12 steps. Step time 
was pseudo-randomized per run (8 s/step, 6s/step or 4s/step). Video clips of the patients’ runs were cut, shuffled 
and given to an expert physiotherapist blinded to the conditions of the experience, with a questionnaire devel- 
oped by our group (referred to here as visual gait score for sFES, see Supplementary Table $4 for details). The 
questionnaire included three control, five high-level and seven kinematics-oriented questions. The answers were 
Likert-type, ranging from 1 to 5. The control questions assessed respectively whether the patient exhibited spas- 
ticity, clear signs of fatigue or reduction of muscle response to the stimulation during the run. If any of the three 
control questions had a score superior or equal to 4, the run was discarded. No runs were removed for any of the 
two sessions used for the tactile-on versus tactile-off experiment with patient P1. One of the runs of patient P2 
and none of P1 were discarded from the qualitative analysis of sFES-generated walk. 


3D Gait analysis. Standard gait-analysis procedures were employed to collect data using a motion-capture sys- 
tem comprised of 12 cameras and a 4 Mb resolution (Raptor-4) and the Cortex 6.0 software (Motion Analysis, 
Santa Rosa, CA, USA). The kinematic data were acquired at 150 Hz. The marker-set protocol adopted was 
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comprised of 28 anatomical reflective markers (Helen Hayes marker set). Markers were placed directly onto 
the skin in the patient’s pelvic and lower-extremity segments. Offline, raw marker-trajectory data were filtered 
using a 4th-order, low-pass Butterworth filter with a cut-off frequency of 10 Hz. Data processing was performed 
using the Cortex software version 6.0. Visual 3D software (C-motion Inc., Germantown, MD, USA) was used to 
perform all kinematics calculations. The foot strike and toe-off were determined when the velocity of the heel 
and toe markers respectively crossed a 20 mm/s threshold level. The definition of the anatomical-segment coor- 
dinate system was used to determine the 3D position and orientation of the lower extremity and pelvis segments 
through a combination of anatomical-frame conventions proposed previously”. Patient P2 performed 54 full gait 
cycles, with 101 time-normalized points, which corresponded to the time-normalized angles (pelvis, hip, knee, 
ankle, and foot). 


Data Availability 
The custom code used for the experiments and the data that support the findings of this study are available from 
the corresponding author upon reasonable request. 
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Stimulation of the vagus nerve has become an effective method 
for desynchronizing the highly coherent neural activity typically 
associated with epileptic seizures. This technique has been used 
in several animal models of seizures as well as in humans suf- 
fering from epilepsy. However, application of this technique has 
been limited to unilateral stimulation of the vagus nerve, typically 
delivered according to a fixed duty cycle, independently of 
whether ongoing seizure activity is present. Here, we report that 
stimulation of another cranial nerve, the trigeminal nerve, can 
also cause cortical and thalamic desynchronization, resulting ina 
reduction of seizure activity in awake rats. Furthermore, we 
demonstrate that providing this stimulation only when seizure 
activity begins results in more effective and safer seizure reduc- 
tion per second of stimulation than with previous methods. Sei- 
zure activity induced by intraperitoneal injection of pentylenetet- 
razole was recorded from microwire electrodes in the thalamus 
and cortex of awake rats while the infraorbital branch of the 
trigeminal nerve was stimulated via a chronically implanted nerve 
cuff electrode. Continuous unilateral stimulation of the trigeminal 


nerve reduced electrographic seizure activity by up to 78%, and 
bilateral trigeminal stimulation was even more effective. Using a 
device that automatically detects seizure activity in real time on 
the basis of multichannel field potential signals, we demon- 
strated that seizure-triggered stimulation was more effective than 
the stimulation protocol involving a fixed duty cycle, in terms of 
the percent seizure reduction per second of stimulation. In con- 
trast to vagus nerve stimulation studies, no substantial cardio- 
vascular side effects were observed by unilateral or bilateral 
stimulation of the trigeminal nerve. These findings suggest that 
trigeminal nerve stimulation is safe in awake rats and should be 
evaluated as a therapy for human seizures. Furthermore, the 
results demonstrate that seizure-triggered trigeminal nerve stim- 
ulation is technically feasible and could be further developed, in 
conjunction with real-time seizure-predicting paradigms, to pre- 
vent seizures and reduce exposure to nerve stimulation. 


Key words: epilepsy; trigeminal nerve; seizure detection; sei- 
zure control; pentylenetetrazole; bilateral stimulation 


Seminal neurophysiological studies performed several decades ago 
demonstrated that stimulation of either cranial nerves or areas of 
the brainstem can cause desynchronization of the cortical EEG 
(Moruzzi and Magoun, 1949; Zanchetti et al., 1952; Magnes et al., 
1961; Chase et al., 1967). Such desynchronization typically reflects 
a state of arousal and full vigilance in mammals and is opposite to 
the high degree of EEG synchronization observed during seizure 
activity. Building on these classical findings, several researchers 
showed that stimulation of the vagus nerve can lead to EEG 
desynchronization (Zanchetti et al., 1952; Chase et al., 1966, 1967; 
Chase and Nakamura, 1968). More recently, several studies have 
demonstrated that the desynchronization induced by vagus nerve 
stimulation (VNS) in dogs can be used to reduce strychnine- or 
pentylenetetrazole (PTZ)-induced seizure activity (Zabara, 1985, 
1992). This paradigm was demonstrated subsequently to be effec- 
tive in other animals, with other seizure models (Lockard et al., 
1990; Woodbury and Woodbury, 1990; McLachlan, 1993), and has 
been used with moderate success in treating humans who have 
otherwise intractable epileptic seizures (Penry and Dean, 1990; 
Uthman et al., 1990, 1993; Ben-Menachem et al., 1994; Vagus 
Nerve Stimulation Study Group, 1995; McLachlan, 1997; Schachter 
and Saper, 1998). Because 0.5-2% of the population has epilepsy 
(Schachter and Saper, 1998; McNamara, 1999) and 10-50% of 
these patients do not respond well to antiepileptic medications 
and/or may not be candidates for resective epilepsy surgery 
(McLachlan, 1997; Schachter and Saper, 1998), there is a substan- 
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tial need for potential alternative therapies for chronic seizures. 
Indeed, the VNS technique has recently received FDA approval 
and is currently being used in patients. 

There are, however, several limiting factors of the VNS tech- 
nique, which, if addressed, could greatly increase the efficacy and 
applicability of cranial nerve stimulation for seizure reduction in 
patients. First, the standard implementation of VNS in humans 
typically involves stimulating the vagus nerve on a fixed, intermit- 
tent duty cycle (e.g., 30 sec on; 5 min off; 24 hr a day), indepen- 
dently of whether any seizure activity is ongoing or imminent 
(although the use of manual patient- or caregiver-triggered stimu- 
lation via a handheld magnet has also been used) (Terry et al., 
1990; Uthman et al., 1993). This type of protocol has been used in 
previous studies for two main reasons. First, although continuous 
stimulation may have a greater therapeutic effect than intermittent 
stimulation (Takaya et al., 1996), continuous stimulation can cause 
nerve damage, whereas intermittent stimulation does not (Agnew 
et al., 1989; Agnew and McCreery, 1990; Ramsay et al., 1994). 
Second, the side effects associated with VNS are typically experi- 
enced during the stimulation (Uthman et al., 1993; Ramsay et al., 
1994; McLachlan, 1997), so giving intermittent stimulation reduces 
their occurrence. However, because stimulation is delivered re- 
gardless of whether seizure activity is present or is likely to occur, 
this fixed stimulation protocol has the disadvantage that the patient 
may receive excess stimulation. 

The second main problem is that the vagus nerve is involved in, 
among other things, cardiovascular and abdominal visceral func- 
tions. Indeed, because of the pattern of vagus innervation of the 
heart, the vagus nerve can only safely be stimulated unilaterally 
(i.e., on the left side only). This is a potential limitation in the 
efficacy of cranial nerve stimulation because the effects of the 
stimulation may be bilateral (Chase et al., 1966; Zabara, 1992; 
Henry et al., 1998, 1999) and may, therefore, be aided by adding 
more stimulation sites. For these reasons, use of a nerve without 
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Figure 1. Schematic drawing of nerve cuff electrode and ASD device. 
Field potential signals from chronically implanted microwires in the VPM 
thalamus and/or SI cortex were sent to an amplifier and recording unit for 
collection, as well as to the ASD device. When the ASD device detected 
seizure activity, it sent a signal to the stimulator, which delivered a current 
pulse to the implanted nerve cuff electrode. Scale bar: inset, 1 mm. 


the types of visceral fibers that are found in the vagus nerve could 
potentially be more effective for seizure reduction. 

Here, we demonstrate that trigeminal nerve stimulation reduced 
PTZ-induced seizures in awake rats. In addition, we show that such 
stimulation was more effective when it was bilateral. Finally, we 
describe a real-time interface for automatically detecting seizure 
activity and eliminating it by providing stimulation only when 
seizure activity is present. These results suggest that if such tech- 
niques were implemented in human patients, they could greatly 
decrease the amount of stimulation necessary for seizure control 
while increasing the efficacy of cranial nerve stimulation as a 
therapy for intractable epilepsy. 


MATERIALS AND METHODS 


Subjects. Eight adult female Long-Evans hooded rats weighing between 
230 and 375 gm served as subjects in this study. All procedures and 
experiments were conducted in compliance with Duke University Medical 
Center animal use policies and were approved by the Duke University 
Institutional Animal Care and Use Committee. 

Induction of seizures. Seizures were induced by intraperitoneal injection 
of PTZ (40 mg/kg). This dose of PTZ induced generalized seizure activity 
for 1-2 hr. This seizure activity was manifested in two ways: (1) highly 
synchronous, large-amplitude activity in the thalamic and cortical field 
potential traces (see Figs. 2, 3, 5-7) and (2) clonic jerking of the body and 
forelimbs. These two indicators of seizure activity were highly correlated 
at all times, as assessed by concurrent visual inspection of the animal and 
the real-time field potential traces. Occasionally, a supplemental dose of 
PTZ (7-10 mg/kg) was given if seizures ceased in <1 hr. 

Nerve cuff electrodes. The infraorbital nerve(s) was stimulated unilater- 
ally or bilaterally via chronically implanted nerve cuff electrodes. These 
electrodes were constructed in-house and consisted of two bands of plat- 
inum (0.5 mm wide and 0.025 mm thick; ~0.8 mm separation between 
bands) that ran circumferentially around the infraorbital (IO) nerve (see 
Fig. 1, inset). The platinum bands were held in place and electrically 
insulated by a thin Sylgard coating. Each band was connected to a piece of 
flexible, three-stranded Teflon-coated wire that was used to pass current 
between the bands (Fanselow and Nicolelis, 1999). 


J. Neurosci., November 1, 2000, 20(21):8160-8168 8161 


Chronic implantation of microwire electrodes. Microwire electrodes 
(NBLabs, Denison, TX) were chronically implanted into the ventral pos- 
terior medial thalamus (VPM) and/or primary somatosensory cortices (SI) 
for use in recording field potentials in these areas (Fig. 1). Three rats had 
arrays of 16 microwires implanted in layer V of the SI cortex and bundles 
of 16 electrodes implanted into the VPM thalamus, both contralateral to 
the stimulated nerve. Five rats had two arrays of 16 electrodes implanted, 
one each into layer V of the left and right SI cortices so that recordings 
could be made both ipsilateral and contralateral to the nerve being 
stimulated. 

These implants were performed under pentobarbital anesthesia (50 
mg/kg). Small craniotomies were performed over the areas into which 
electrodes were to be implanted [coordinates from Paxinos and Watson 
(1986)]. Electrodes were slowly lowered into these areas, and recordings 
were made throughout the implantation process to assess electrode loca- 
tion. After electrodes were in the correct position, they were cemented to 
skull screws by the use of dental acrylic (Nicolelis et al., 1997). 

Chronic implantation of nerve cuff electrodes. After implantation of the 
microwires, nerve cuff electrodes were implanted either unilaterally or 
bilaterally. A dorsoventral incision was made on the face several millime- 
ters caudal to the caudal edge of the whiskerpad. Tissue was dissected until 
the infraorbital nerve was exposed, and the cuff electrode was positioned 
around the nerve such that the nerve lay inside the cuff. The cuff was then 
tied around the nerve to hold it in place, and the wound was sutured. The 
Teflon-coated leads from the platinum bands were run subcutaneously to 
the top of the head where they were attached to connector pins and affixed 
to the skull. 

Recording procedures. Field potential recordings from VPM thalamus 
and SI cortex were made using chronically implanted microwires (Nicolelis 
et al., 1997). Field potentials were collected using a Grass Model 15 
amplifier and stored on a personal computer. Signals were collected at a 
sampling rate of 512 Hz and bandpass filtered during collection at 1-100 
Hz. During each recording session, 16 field potential channels were re- 
corded, 8 from each area from which recordings were made in a given rat 
(either VPM and SI, or SI left and SI right). In addition, one channel was 
recorded for each nerve cuff being stimulated (unilateral or bilateral 
stimulation) to indicate when stimulation occurred. During experiments, 
animals were awake and allowed to move freely in a 30 cm X 30 cm 
recording chamber. 

Stimulation parameters. Stimulation of the IO nerve cuff electrodes was 
provided by the use of a Grass $8800 stimulator in conjunction with a Grass 
SIU6 stimulus isolation unit. Unimodal square current pulses with a 
duration of 500 psec were given at a range of currents and frequencies. 
Current values were varied from 3 to 11 mA (2 mA intervals), and 
frequency values were varied from 1 to 333 Hz (1, 5, 10, 20, 50, 100, 125, 
200, and 333 Hz). Animals tolerated stimulation at these levels without 
indication of pain, although in some animals there appeared to be a 
sensation of pressure on the face at the highest current and frequency 
settings. This was evidenced by a tendency for the animals to back up when 
the stimulus began, in the direction away from the stimulated side if 
unilateral stimulation was provided or straight back in the case of bilateral 
stimulation. In addition, at lower stimulus intensities animals would occa- 
sionally scratch at the whiskerpad on the side of the face being stimulated 
during the first few seconds of stimulation. However, the scratching was 
neither intense nor prolonged. 

Automatic seizure detection device. A device was designed and built 
in-house to automatically detect seizure activity in real time and immedi- 
ately trigger a stimulator when a seizure was detected (Fig. 1). The 
automatic seizure detection (ASD) device first low-pass filtered the raw 
field potentials obtained from the microwire arrays at 30 Hz. Circuitry 
then determined whether the field potential activity surpassed a threshold 
voltage value, indicative that seizure activity was present. When the field 
potential voltage crossed the threshold, a TTL pulse was sent to the Grass 
$8800 stimulator, which delivered a 0.5 sec train of 500 psec pulses at 333 
Hz. The current level was dictated by the stimulation protocol for a given 
trial. Trains of stimuli were presented as long as the field potential activity 
remained above the threshold value (i.e., as long as seizure activity was 
ongoing). The train duration for the seizure-triggered stimulation (0.5 sec) 
was chosen because it was the shortest duration that we found to be 
effective for stopping the seizure activity, and we wanted to keep the 
stimulation as short as possible to reduce the total amount of stimulation 
given. The voltage threshold was set manually for each experiment. Gen- 
erally, the seizure activity was three to five times that of the background 
activity, and the threshold was set high enough to identify seizure activity 
only. After the threshold was set for a given experiment, it was not moved. 
The seizure activity recorded on the field potential traces was directly 
correlated with behavioral manifestation of the seizures (clonic jerking of 
the body and forelimbs). When setting the seizure detection threshold, we 
always verified that the seizure activity identified by the ASD device was 
directly correlated with this behavioral component of the seizures. 

Experimental protocols. The first part of this study was performed to 
determine whether stimulation of the IO branch of the trigeminal nerve 
was capable of eliminating PTZ-induced seizure activity in awake rats. To 
do this, we delivered continuous stimulation to the IO nerve during 
episodes of PTZ-induced seizure activity via the nerve cuff electrode (Fig. 
1) for 1 min stimulus-on periods, separated by 1 min stimulus-off periods. 
This protocol was performed with both unilateral and bilateral stimulation 
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of the IO nerve. Stimulus parameters were varied between the stimulus-on 
periods as described above. 

In the second part of this study, we assessed the effectiveness of stimu- 
lating the IO nerve only when seizure activity was present by using the 
ASD. For this protocol, the ASD device was turned on for 1 min 
stimulus-on periods, separated by 1 min stimulus-off periods, as in the first 
protocol, but stimulation was only provided during the stimulus-on periods 
when seizure activity was detected by the ASD device. 

Data analysis. We measured seizure activity in the field potential re- 
cordings in three ways: seizure frequency, seizure duration, and integrated 
seizure activity. These parameters were quantified by the use of a custom- 
made analysis program developed using Matlab. The field potential traces 
were first bandpass filtered at 5-30 Hz. A sliding window (1 sec window 
with 0.5 sec overlap) was used to quantify the activity of the absolute values 
of the field potential traces. Within each window, the amplitude (i.e., 
voltage) range of the absolute value of the field potential activity in each 
trace was divided into 10 equal parts, and within each sliding window the 
number of voltage values falling into each of the 10 divisions of the 
amplitude range was calculated. Then, a threshold of 50% of the amplitude 
range was used to identify seizure activity. If activity within three consec- 
utive windows was above this threshold, the activity was considered to be 
part of a seizure. From these data, the number of seizures and their 
durations could be calculated by counting the number of windows during 
which activity was above the threshold. In addition, a measure we call the 
“integrated seizure activity” was calculated by summing all of the values 
for all of the amplitude range intervals for a given on or off period of 
stimulation. This algorithm was applied in a uniform, blinded manner to all 
of our data, allowing for objective quantification of the three measures of 
seizure activity. 

Statistical analyses. Using the values for seizure number, seizure dura- 
tion, and integrated seizure activity, we assessed the efficacy of IO nerve 
stimulation and ASD by comparing each stimulation-on period with the 
stimulation-off period directly preceding it. Thus, results are presented as 
ratios of seizure activity during stimulus-on periods to seizure activity 
during stimulus-off periods. We used multivariate ANOVAs (MANOVAs) 
to assess whether there were statistically significant changes in seizure 
duration, seizure frequency, or integrated seizure activity between periods 
of no stimulation and periods of stimulation for each stimulus parameter 
setting. In addition, repeated measure ANOVAs were used when com- 
paring one measure with another (e.g., number of seizures compared 
with seizure duration). When significant differences were indicated by 
MANOVA or ANOVA analyses, Tukey’s honestly significant difference 
post hoc tests were used to identify which effects were significant (p < 
0.05). 


RESULTS 


Control experiments 


In control experiments in which PTZ was administered, but no IO 
nerve stimulation was provided, the average number of seizures per 
minute was 5.98 + 0.45, and the average seizure duration was 
3.94 + 0.23 sec. 

In contrast to studies of VNS in rats (Woodbury and Woodbury, 
1990) and dogs (Zabara, 1992), we did not observe any substantial 
cardiovascular side effects during IO nerve stimulation (Fig. 2). We 
recorded electrocardiogram (EKG) signals in anesthetized rats 
while stimulating the IO nerve and did not observe any substantial 
change in heart rate during stimulation. 


Stimulation of the infraorbital nerve reduces 
seizure activity 


Stimulation of the infraorbital nerve by the use of the periodic 
stimulation paradigm substantially reduced PTZ-induced seizure 
activity compared with that of control periods (Figs. 3, 4, 5). This 
effect was dependent on both the current and the frequency of the 
stimulation. There were no significant differences between the 
cortex and thalamus on any of the measures [Rao R(3,134) = 0.33; 
p > 0.8]. 

As expected from previous studies, the seizure reduction effect 
of IO nerve stimulation was greater with increasing current levels 
(Fig. 3b-d). For these experiments, pulse duration and frequency 
were held constant at 0.5 msec and 333 Hz, respectively, while 
current was varied between 3 and 11 mA, in 2 mA increments. At 
currents of 3 and 5 mA, there were no differences between periods 
of IO nerve stimulation and periods of no stimulation. However, at 
7, 9, and 11 mA, nerve stimulation caused a significant decrease in 
overall seizure activity (Fig. 3b, 7 mA, 43.2 + 7.0%; 9 mA, 65.5 + 
4.7%; 11 mA, 77.5 + 4.3%; p < 0.001) and in the number of 
seizures initiated (Fig. 3c, 7 mA, 36.4 + 5.8%; 9 mA, 50.5 + 4.6%; 
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Figure 2. EKG activity is not significantly altered during IO nerve stim- 
ulation. a, b, Two examples of EKG activity during IO nerve stimulation 
(stim; horizontal bars) in an anesthetized rat. Calibration: vertical, 100 »V; 
horizontal, 1 sec. c, The EKG traces and instantaneous heart rate (Inst. rate) 
over a 15 min period during which stimulation was twice provided contin- 
uously for 1 min, as well as five times for shorter bursts. Small changes in the 
EKG can be seen when stimulation is provided, but they are minor and 
rapidly stabilize, even during ongoing stimulation. i, ii, The traces that are 
shown at a faster time scale in a and b, respectively. Calibration: vertical, 
100 pV for the EKG traces, 200 beats/min for the instantaneous heart rate; 
horizontal, 10 sec. The stimulus parameters in these traces were 50 Hz, 11 
mA, and 0.5 msec pulse duration. 


11 mA, 58.7 + 6%; p < 0.0001). There was also a significant 
decrease in the seizure duration at 9 mA (Fig. 3d, 52.5 + 3.7%; p < 
0.0001). 

Different stimulus frequencies had different effects on the seizure 
activity (Fig. 4). For these experiments, pulse duration and current 
were held constant at 0.5 msec and 9 mA, respectively. Stimulation 
at high frequencies (100, 125, 200, and 333 Hz) caused a signifi- 
cantly smaller number of seizures than did periods of no stimula- 
tion (Fig. 4a; p < 0.05), as described above. Stimulation frequencies 
of 50 Hz and lower did not cause any significant changes in the 
number of seizures initiated (Fig. 4a; p = 1.0), but seizures did tend 


119 


Fanselow et al. ¢ Seizure Reduction by Trigeminal Nerve Stimulation 


J. Neurosci., November 1, 2000, 20(21):8160-8168 8163 


» Db Integrated seizure activity 


Seizure duration 


VPM[ 


si[ 


no stim 


% of average stimulus-off period values 


3.95 7 9 11 


Current (mA) 


Figure 3. Stimulation of the IO nerve reduces seizure activity in a current-dependent manner. aJ—a3, Filtered field potential traces showing seizure 
activity during three sequential 1 min periods (a/, no stimulus; a2, stimulus on; a3, no stimulus). The stimulus parameters for this figure were 11 mA, 333 
Hz, and 0.5 msec pulse. b—d, The amount of seizure activity during 1 min periods of stimulation at different current levels compared with the period of 
no stimulation directly preceding each stimulus-on period. Values are presented as a percent of the average stimulus-off period measurements. b, 
Integrated seizure activity. c, Number of seizures. d, Seizure duration. Error bars represent +SEM. A solid line connects stimulation-off values; a dashed 
line connects stimulation-on values. Stimulation-on values significantly different from stimulation-off values are designated by an asterisk. Thick, black 
horizontal lines at 100% denote the level of no change in seizure activity. Calibration: vertical, 200 wV; horizontal, 10 sec. 
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Figure 4. Effect of varying stimulus frequency using the periodic stimu- 
lation paradigm. a, Number of seizures. b, Seizure duration. Labeling 
conventions are described in Figure 3 (note the change in the scale of the 
y-axis in b). 


to be longer than those during control periods at these frequencies 
(Fig. 4b; 10 Hz; p < 0.02). 


Bilateral versus unilateral stimulation 


Bilateral stimulation was significantly more effective at reducing 
seizures than was unilateral stimulation either contralateral or 
ipsilateral to the recording site (Fig. 5). This effect was significant 
for the integrated seizure activity measure (Fig. 5b) at a current 
level of 7 mA (75.7 £ 5.7%; p < 0.002), as well as for the number 
of seizures (Fig. 5c) at 7 and 9 mA (7 mA, 63.7 + 5.3; 9 mA, 78.1 + 
3.7%; p < 0.01). It is important to point out that the superior effect 
of bilateral stimulation was only evident for the middle range of 
stimulation intensities used in this study. That is, if the current was 
too low, presumably below the threshold for seizure reduction, 
there was no advantage in stimulating both nerves, and if the 


current was high enough, stimulating unilaterally was as effective as 
stimulating bilaterally. However in the middle range of stimulation 
intensities, bilateral stimulation allowed us to use less current per 
nerve while still maintaining a high degree of seizure reduction. 


Automatic detection of seizure activity and termination 
of seizures 


Use of the ASD device to stimulate the IO nerve only when seizure 
activity was detected successfully reduced the amount of seizure 
activity relative to control periods. Figure 6 shows that when the 
seizure detector identified seizure activity in the field potential 
traces and triggered the stimulator, the seizure stopped. As in the 
experiments described above, the degree of seizure reduction was 
dependent on the current level (Fig. 7). For this set of experiments, 
we held the pulse duration constant at 0.5 msec, and the frequency 
at 333 Hz. Current was varied from 3 to 11 mA in 2 mA increments. 
Figure 7b shows that the integrated seizure activity level was 
significantly reduced at 9 and 11 mA (9 mA, 55.2 = 7.2%; p < 0.03; 
11 mA, 56.6 + 8.0%; p < 0.01). The number of seizures was 
significantly decreased at 7 and 9 mA (Fig. 7c, 7 mA, 19.3 + 5.8%; 
p <.0.05; 9 mA, 22.5 + 6.1%; p < 0.0001). In addition, the seizure 
duration was decreased at 7, 9, and 11 mA (Fig. 7d, 7 mA, 40.2 + 
3.3%; 9 mA, 45.2 + 3.6%; 11 mA, 49.4 + 4.0%; p < 0.0001 for all). 
To compare the efficacy of the ASD device with that of the periodic 
stimulation paradigm, we calculated the ratio of the percent of 
seizure reduction to stimulus-on time (Fig. 8). By comparing these 
ratios between ASD stimulation and periodic stimulation proto- 
cols, we observed that, at least in the acute seizure model (PTZ) 
used in this study, delivering stimulation only when seizure activity 
was detected was up to 39.8 times more effective at seizure reduc- 
tion per second of stimulation than was periodic stimulation not 
related in any way to seizure activity. 

There was an important difference between the nature of the 
seizure reduction effect using the ASD device and that observed 
using the periodic stimulation paradigm described above. With the 
periodic stimulation paradigm, the number of seizures and the 
seizure durations were reduced by approximately the same amount 
at each current level (Fig. 3, compare c, d). However, when the 
ASD device was used, the seizure durations were reduced signifi- 
cantly more than the number of seizures (Fig. 7, compare c, d; p < 
0.000001). 

In addition, analysis of the data revealed that in control experi- 
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Figure 5. Effects of bilateral stimulation versus unilateral stimulation. al—a3, Filtered field potential traces showing seizure activity during three 
sequential 1 min periods (a/, no stimulus; a2, bilateral stimulation; a3, no stimulus). The stimulus parameters were 9 mA, 333 Hz, and 0.5 msec pulse 
duration. b—d, Values presented as ratios of stimulus-on/stimulus-off measurements. b, Integrated seizure activity. c, Number of seizures. d, Seizure 
duration. A solid line connects responses contralateral to the stimulation site; a line with long dashes connects responses ipsilateral to the stimulation site; 
a line with short dashes connects responses to bilateral stimulation. Responses to bilateral stimulation that are significantly different from those to ipsilateral 
and contralateral stimulation are represented by an asterisk. Other labeling conventions are described in Figure 3. 


a 


SI field 
potential 


stimulation 


seizure 
detector 


b 
SI field 
potential 


stimulation 


seizure 
detector 


SI field 
potential 


stimulation 


seizure 
detector 


Figure 6. Seizure-specific stimulation stops synchronous activity. When 
the field potential amplitude reached a threshold value, the seizure detector 
(ASD device) triggered IO nerve stimulation. a—c, Traces correspond to 
the roman numerals i-iii, respectively (see Fig. 7). Note that the stimulation 
outlasts seizure activity because pulses were provided in 500 msec trains. 
Also note that the traces indicating seizure detection and stimulation are 
only indicators and are not indicative of stimulation current or frequency. 
Calibration: vertical, 200 pV; horizontal, 500 msec. 


ments where PTZ was administered but no stimulation was pro- 
vided, the average time between the end of one spontaneously 
occurring seizure and the beginning of the next was 6.1 sec (cal- 
culated from the average number of seizures and the average 
seizure duration). We also measured the latency between the end 


of a stimulus and the next spontaneous seizure (i.e., in the epoch 
after a stimulus-on period), which was 7.59 + 1.29 sec. Thus, the 
average delay between the end of a period of stimulation and the 
next spontaneously occurring seizure is an average of 24% longer 
than the interseizure interval during control experiments where no 
stimulation was present. 


DISCUSSION 


The results of this study demonstrate three substantial advances in 
the use of cranial nerve stimulation for the treatment of seizures. 
First, we showed that stimulation of the trigeminal nerve can 
reduce PTZ-induced seizure activity in rats. This indicates that the 
seizure reduction effect of cranial nerve stimulation is not limited 
to stimulation of the vagus nerve but instead may be mediated by a 
more nonspecific arousal mechanism that can be recruited by 
stimulation of a number of cranial nerves. Second, we showed that 
bilateral trigeminal nerve stimulation could have the same seizure 
reduction effect as unilateral stimulation but required much less 
current to do so. This finding is therapeutically relevant, because it 
suggests that multisite stimulation could help maximize the seizure 
reduction effect of any technique using cranial nerve stimulation, 
while using the lowest current levels possible. Finally, we showed 
that in the acute seizure model used in this study (PTZ), automatic, 
real-time seizure-triggered stimulation reduces seizures more ef- 
fectively per second of stimulation than does periodic stimulation 
that is unrelated to seizure onset. This means that the use of a 
real-time brain—device interface that would automatically detect 
seizure activity and trigger a nerve stimulator only when such 
activity was present could provide a high degree of seizure control 
while potentially reducing the overall amount of stimulation pre- 
sented to a patient. We propose that these findings may signifi- 
cantly improve the efficacy of cranial nerve stimulation as a therapy 
for patients with intractable epileptic seizures. 


Mechanism of seizure reduction by cranial 

nerve stimulation 

The mechanism by which cranial nerve stimulation causes desyn- 
chronization of thalamic and cortical activity and reduces seizure 
activity is unknown. However, one theory is that such stimulation 
activates the midbrain reticular formation and that this activation 
results in generalized arousal via the reticular-activating system. In 
support of this view, Gellhorn (1960) showed that stimulation of 
the midbrain reticular formation suppresses focal strychnine spikes 
in cats. In addition, several methods of eliminating seizure-related 
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Figure 7. Seizure reduction using the ASD device. al—a3, Filtered field potential traces showing seizure activity during three sequential 1 min periods 
(al, no stimulus; a2, stimulus on; a3, no stimulus). The stimulus parameters were 9 mA, 333 Hz, and 0.5 msec pulse duration. Within each segment, the 
trace labeled seizure detector indicates where the ASD device detected seizure activity; the trace labeled stimulus on indicates where the ASD device sent 
a TTL pulse to trigger the IO nerve stimulator when it detected such activity. The roman numerals i-iii and arrows indicate parts of the traces that were 
enlarged to create Figure 6. b, Integrated seizure activity. c, Number of seizures. d, Seizure duration. Labeling conventions are described in Figure 3. 
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Figure 8. Comparison of the amount of seizure reduction versus the 
amount of stimulation provided. Stimulation was provided by the use of the 
periodic stimulation paradigm (dashed line) or the ASD device (solid line). 
The y-axis represents the ratio of seizure activity reduction to seconds of 
stimulation in a given stimulus-on period. Asterisks designate the ratios of 
ASD seizure reduction to seconds of stimulation that were significantly 
higher than those obtained by the use of the periodic stimulation protocol. 
a, Integrated seizure activity. b, Number of seizures. c, Seizure duration. 


activity by activating multiple sensory modalities have been dem- 
onstrated. These include the reduction of absence seizures by 
acoustic stimuli (Rajna and Lona, 1989) and the reduction of 
interictal focal activity or absence seizures by motor or mental 
activity (Jung, 1962; Ricci et al., 1972) or by thermal stimulation 
(McLachlan, 1993). Because such a wide range of manipulations 
can reduce seizure-related activity, it is reasonable to suggest that 
seizure reduction in these cases is caused by a generalized effect on 
arousal mediated by the brainstem reticular formation. This is 
supported by the classical work of Moruzzi and Magoun (1949) 


demonstrating that stimulation of the midbrain reticular formation 
causes EEG desynchronization. This hypothesis is consistent with 
our finding that seizure reduction effects are not specific to the 
vagus nerve but can instead be achieved by stimulation of multiple 
cranial nerves that convey information to the reticular formation. 

One important factor to consider with regard to both the mech- 
anism of seizure reduction by trigeminal nerve stimulation and its 
applicability to long-term use in humans is the nature of the fiber 
types that must be activated to cause the seizure reduction effect. 
Multiple studies of the VNS technique have shown that the level of 
stimulation, in terms of stimulus frequency and intensity, must be 
high enough to activate slowly conducting c-fibers (Chase et al., 
1967; Woodbury and Woodbury, 1990). The frequency range we 
found to be therapeutic in the present study was somewhat different 
from that typically used in animal and human VNS studies. In 
animal studies the usual therapeutic range was generally 10-30 Hz 
(Woodbury and Woodbury, 1990; Zabara, 1992; Takaya et al., 
1996), although Lockard et al. (1990) used higher stimulation 
frequencies (50-250 Hz) in monkeys. In human studies the range 
used for stimulation was typically 20-30 Hz (McLachlan, 1997). 
This difference between VNS studies and ours may be caused by 
the difference in the relative numbers of fiber types between the 
vagus nerve and the infraorbital nerve. In cat, the vagus nerve is 
composed of 65-90% unmyelinated fibers (Foley and DuBois, 
1937; Agostoni et al., 1957), whereas the rat IO nerve contains 
~33% slowly conducting, unmyelinated fibers (Klein et al., 1988). 
However, it is not clear what the relationship is between fiber 
composition and the stimulus frequency/intensity required for sei- 
zure reduction, so interpreting these differences is difficult. A 
complicating factor is that although it has been shown that for 
seizure reduction the level of stimulation must be sufficient to 
activate c-fibers, it has not been demonstrated that these fibers are 
necessary for the seizure reduction effect. Finally, it is important to 
note that according to studies by Torebjork and colleagues (Tore- 
bjork, 1974; Torebjork and Hallin, 1974), c-fibers do not conduct if 
electrical stimuli are presented at frequencies above ~10 Hz. This 
means that although high stimulation frequencies were required for 
the seizure reduction effect observed here, it is likely that at such 
frequencies the c-fibers were not activated or were activated to a 
lesser degree than other fibers in the nerve. 

Furthermore, it is possible that cells in the trigeminal nucleus 
were not able to follow with sustained responses at the high rates of 
stimulation we provided. For example, Andresen and Yang (1995) 
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demonstrated using a slice preparation of the rat medulla that 
neurons in the nucleus of the solitary tract (NTS) responded with 
lower EPSP amplitudes as the frequency of solitary tract stimula- 
tion was increased. These results may also be relevant to trigeminal 
nerve stimulation. If this is the case, it is unclear why our results 
show that higher frequency stimulation is more effective for seizure 
elimination than are lower stimulation rates. However, the study by 
Andresen and Yang (1995) also demonstrated that bursts of high- 
frequency stimulation resulted in less EPSP attenuation than did 
continuous high-frequency stimulation, suggesting that an optimal 
stimulation protocol could involve short bursts of high-frequency 
stimulation rather than continuous trains. 

The delay between the onset of seizure-triggered stimulation 
and the end of the seizure activity might shed some light on the 
mechanism by which trigeminal stimulation reduces seizure activ- 
ity. The average time between the onset of the seizure-triggered 
stimulus and the end of the seizure was 529.9 + 40.3 msec (note 
that Fig. 6 demonstrates some of the shortest delays). It is inter- 
esting that this value is similar to the minimum effective stimulus 
train duration (500 msec) that we determined empirically. How- 
ever, it should be noted that there was a wide range of delays, and 
this may be caused by at least two factors. First, the phase of the 
synchronous oscillations during which the stimuli occur may have a 
profound impact on the efficacy of the stimulation. Second, it is 
possible that the ability to abort a seizure varies depending on the 
amount of time the seizure has been ongoing before a sufficient 
stimulus arrives. Thus, differences in the phase of the oscillatory 
seizure activity at which the stimuli occur or the threshold used for 
seizure detection may affect the efficacy of the stimulation. These 
mechanisms could explain the variation in the amount of time 
required to abort a seizure. 

Another important mechanism-related issue is whether the tri- 
geminal stimulation was merely able to stop seizure activity during 
the stimulation itself or whether it also had an effect on the number 
of seizures initiated. In control files where PTZ was administered 
but no stimulation was provided, the average time between the end 
of one spontaneously occurring seizure and the beginning of the 
next was 6.1 sec (calculated from the average number of seizures 
and the average seizure duration, as reported in Results). We also 
measured the latency between the end of a period of stimulation 
and the next spontaneous seizure (i.e., in the epoch after a 
stimulus-on period), which was 7.59 + 1.29 sec. Thus, the average 
delay between the end of a period of stimulation and the first 
spontaneous seizure after the stimulus ends is actually, on average, 
24% longer than the interseizure interval during control files with 
no stimulation present. These results are supported by results from 
other laboratories (Zabara, 1992; Takaya et al., 1996) showing that 
the seizure reduction effect of vagus nerve stimulation can outlast 
the stimulus duration. 


Bilateral versus unilateral |O nerve stimulation 


The fact that bilateral stimulation can be more effective than 
unilateral stimulation in the middle of the therapeutic-current 
range has implications for how such stimulation could be used to 
most effectively reduce seizure activity. Specifically, because bilat- 
eral stimulation at 7 mA was just as effective as unilateral stimula- 
tion at 11 mA (Fig. 5), the use of bilateral nerve cuff electrodes 
would reduce the amount of current delivered to each nerve, while 
still maintaining the same seizure reduction effect as higher stim- 
ulation current at a single site. This would be beneficial because it 
would reduce the potential for damage to nerve fibers at the 
stimulation site (Agnew et al., 1989; Agnew and McCreery, 1990), 
and it would reduce the intensity of any possible side effects 
associated with the stimulation. Bilateral stimulation is a further 
improvement over VNS, because the vagus nerve cannot be safely 
stimulated bilaterally without substantial risk of cardiovascular side 
effects (Schachter and Saper, 1998). 

It is important to point out that our finding that bilateral stim- 
ulation of the IO nerve was more effective than unilateral stimu- 
lation is in contrast to two previous studies reporting that bilateral 
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stimulation of the vagus nerve was no more effective than unilateral 
stimulation (Chase et al., 1966; Zabara, 1992). This discrepancy is 
likely either caused by differences in fiber composition between the 
vagus nerve and the IO nerve or caused by the fact that the stimulus 
parameters used in those studies were beyond those for which 
bilateral stimulation is superior to unilateral stimulation. Details 
about the stimulus parameters used for assessing the efficacy of 
bilateral stimulation in those two studies were not provided. 

Another important point to consider is that we have tested the 
effect of bilateral stimulation with the PTZ seizure model, which 
involves generalized, tonic-clonic seizures (Fisher, 1989). Further 
testing with focal seizure models such as localized application of 
alumina gel (Lockard et al., 1990) or penicillin (McLachlan, 1993) 
to the cortex will be necessary to determine whether there is an 
advantage to bilateral stimulation in eliminating these types of 
seizures as well. Evidence to support an advantage in using bilat- 
eral stimulation to treat focal seizures is that, in our study, unilat- 
eral stimulation eliminated seizure activity in both hemispheres at 
the same time, suggesting that the effect of the stimulation is not 
restricted to one hemisphere. Such results have also been found 
for VNS in cats (Chase et al., 1966), dogs (Zabara, 1992), and 
humans (Henry et al., 1998, 1999). These results suggest that 
because each nerve being stimulated can reduce seizures bilater- 
ally, the effect of stimulating both nerves could be additive within 
a given hemisphere. 


A brain-device interface for automatic, real-time 
detection and reduction of seizure activity 


This study showed that triggering trigeminal nerve stimulation only 
when a seizure began is a much more effective method for reducing 
seizure activity than is providing stimulation on a fixed duty cycle, 
as has been used in past studies. This finding is an important 
advancement in the use of cranial nerve stimulation therapies in 
epilepsy for several reasons. 

First, stimulating only when seizure activity occurs would, for 
many patients, reduce the overall amount of stimulation required 
for maintaining seizure control. Thus, the amount of potentially 
unnecessary stimulation usually occurring between seizure periods 
would be reduced, decreasing the possibility of damage to the nerve 
(Agnew et al., 1989; Agnew and McCreery, 1990; Ramsay et al., 
1994). It is important to note, however, that several researchers 
have demonstrated a prophylactic effect of vagus nerve stimulation 
such that after stimulation, seizures are less likely for a period of 
time related to the duration of the preceding stimulation (Zabara, 
1992; Takaya et al., 1996). This implies that perhaps the best overall 
treatment stimulation protocol might involve the use of seizure- 
triggered stimulation combined with intermittent prophylactic 
nonseizure-triggered stimulation. 

The second advantage of this technique is that it would reduce 
the side effects experienced by patients when the stimulus is on. For 
example, patients undergoing VNS treatment report hoarseness, 
coughing, and throat pain as the most common side effects of the 
stimulation (Ramsay et al., 1994; McLachlan, 1997; Schachter and 
Saper, 1998). These side effects are generally only experienced 
when the stimulation is on. However, if stimulation were only 
presented in response to the detection of seizure activity (or 
occasionally prophylactically, as described above), these side effects 
would be experienced as infrequently as possible. 

In the future, the type of real-time, automatic seizure detector 
described here could be implemented in humans by building on and 
combining a number of existing technologies. First, seizure detec- 
tion could be performed by a computer microchip programmed 
with a seizure detection algorithm and carried by the patient, 
similar to the Holter monitors used for continuous EKG monitor- 
ing. Input would be delivered to this microchip from multiple scalp 
EEG electrodes that would be able to pick up and amplify extracra- 
nial EEG signals. Finally, when the microchip detected seizure 
activity in the EEG signals, it would trigger an implanted stimula- 
tor similar to those used in the VNS technique (Terry et al., 1990), 
which would stimulate one or more trigeminal nerve cuff elec- 
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trodes. This device would function in a manner analogous to 
cardiac pacemakers, commonly used to treat heart arrhythmia, and 
would require a minimum of invasive procedures. In essence, this 
device would constitute a “brain pacemaker” for seizure monitor- 
ing and control. 

The application of nonlinear computational methods for detec- 
tion of seizure activity (Gabor et al., 1996; Webber et al., 1996) 
could be extremely beneficial if incorporated into the seizure 
detector described here. Such seizure detection algorithms would 
allow for more accurate identification of seizure activity than the 
rather simple amplitude-based algorithm we used in this study. 

Another substantial advance could be in the implementation of 
seizure prediction algorithms that can identify seizures seconds or 
minutes before the behavioral onset (Martinerie et al., 1998; Le 
Van Quyen et al., 1999). There is evidence that the sooner stimu- 
lation is provided after a seizure begins, the more effectively the 
seizure can be stopped (Uthman et al., 1993); also stimulation is 
more likely to prevent seizure activity if it is presented before 
rather than after a seizure has begun (Woodbury and Woodbury, 
1990). Therefore, it is possible that providing stimulation before 
the clinically defined onset of a seizure may prevent seizures before 
they begin or become behaviorally relevant to the patient. Such a 
technique could dramatically improve the efficacy of the cranial 
nerve stimulation therapy. 


Application to human patients 


Further studies in other animals and with other seizure models will 
be needed to determine whether it is appropriate to apply the 
techniques described here to human patients and what the best 
methods for doing so would be. 

The results presented here apply to seizures caused by systemic 
administration of PTZ, which is a model of acute, generalized, 
tonic-clonic seizures. It will be important to determine whether the 
results also apply to chronic seizure models, as well as to other 
seizure types, such as focal seizures (e.g., temporal lobe seizures) 
and absence seizures, if the described techniques are to be appli- 
cable to humans. 

Because the cellular mechanisms involved in different seizure 
types may not necessarily be similar, it is vital to ask whether the 
seizure reduction effects from trigeminal nerve stimulation would 
apply to different seizure types. If the effects of trigeminal nerve 
stimulation are spatially restricted or only affect certain types of 
cellular excitation or inhibition, then this technique may be of 
limited use in treating multiple seizure types. If, however, as 
proposed in the current paper, the mechanism by which trigeminal 
nerve stimulation reduces seizure activity is, indeed, a generalized, 
widespread effect on cortical arousal level, perhaps mediated by the 
brainstem reticular-activating system, it is possible that this tech- 
nique would be useful in treating a wide range of seizures. In 
support of this view, the VNS technique has proven to be effective 
in multiple seizure models including intraperitoneal injection of 
PTZ (Zabara, 1985, 1992; Woodbury and Woodbury, 1990), intra- 
peritoneal injection of 3-mercaptopropionate (Woodbury and 
Woodbury, 1990), maximal electroshock (Woodbury and Wood- 
bury, 1990), topical application of penicillin to the cortex 
(McLachlan, 1993), and chronic local application of alumina gel 
(Lockard et al., 1990). 

Another issue that will need to be addressed before this tech- 
nique is applied to humans is that because the trigeminal nerve is 
involved in transmitting both somatosensory and pain information 
from the head, it is vital that the level of stimulation be below that 
that might cause discomfort such as facial pain or headaches. It is 
not known what stimulus parameters would be required to achieve 
seizure reduction without resulting in painful sensations such as 
these. However, such side effects could be substantially reduced by 
using the lowest effective stimulus parameters, which would be 
aided by the use of bilateral stimulation. 

Finally, it would also be possible to develop an effective therapy 
by combining the VNS technique, which is currently in use in 
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human patients, with the automatic seizure detection technique 
described in this paper. 


Conclusions 


The results described in this study could serve to substantially 
increase the efficacy of cranial nerve stimulation as a technique for 
reducing or eliminating seizures in patients who suffer from intrac- 
table epilepsy. Further development and testing of trigeminal nerve 
stimulation for patients with epilepsy is justified on the basis of the 
results presented here. In addition, our findings suggest that in the 
future, it will be feasible to develop a completely implantable and 
relatively noninvasive brain—device interface capable of automat- 
ically detecting seizure activity and triggering stimulation of cra- 
nial nerves to safely and efficiently reduce seizure activity. 
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and reversal of this negative hydrostatic pressure 
to a positive value would result in a corresponding 
pressure-induced expansion in the length direction 
for positive hydrostatic pressure. 

Application possibilities. In addition to extend- 
ing the capabilities of artificial muscles to giant 
strokes and strain rates at extreme temperatures, 
the present actuator mechanism provides other 
application possibilities that relate to the structural 
change of the nanotube sheets during large-stroke 
actuation. The nanotubes diffract light perpendicu- 
lar to the alignment direction, which can be dy- 
namically modulated at over kilohertz frequencies 
for optical applications (74) (movie S1). The ability 
to tune the density of nanotube sheets and then 
freeze this actuation is being used for optimizing 
nanotube electrodes for organic light-emitting dis- 
plays, solar cells, charge stripping from ion beams, 
and cold electron field emission. 
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Spinal Cord Stimulation Restores 
Locomotion in Animal Models 
of Parkinson's Disease 


Romulo Fuentes,?*t Per Petersson,”2* William B. Siesser,? 


Marc G. Caron,*? Miguel A. L. Nicolelist*4>"*7: 


Dopamine replacement therapy is useful for treating motor symptoms in the early phase of 
Parkinson's disease, but it is less effective in the long term. Electrical deep-brain stimulation 

is a valuable complement to pharmacological treatment but involves a highly invasive surgical 
procedure. We found that epidural electrical stimulation of the dorsal columns in the spinal cord 
restores locomotion in both acute pharmacologically induced dopamine-depleted mice and in 
chronic 6-hydroxydopamine—lesioned rats. The functional recovery was paralleled by a disruption 
of aberrant low-frequency synchronous corticostriatal oscillations, leading to the emergence of 
neuronal activity patterns that resemble the state normally preceding spontaneous initiation of 
locomotion. We propose that dorsal column stimulation might become an efficient and less 
invasive alternative for treatment of Parkinson's disease in the future. 


(PD) experience chronic and progressive 
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degeneration of neurons in the dopaminergic ni- 
grostriatal pathway (2). Dopamine replacement 
therapy, through administration of the dopa- 
mine precursor 3,4-dihydroxy-L-phenylalanine 
(L-dopa), effectively ameliorates symptoms as- 
sociated with PD and remains the treatment of 
choice to date (3). Unfortunately, L-dopa phar- 
macotherapy has proven less efficient in the long 
term and is associated with several complications 
(4). Additional therapeutic strategies, employed in 
conjunction with pharmacological treatment, have 
thus attracted considerable attention. In particular, 
improved techniques for electrical stimulation of 
the basal ganglia—treferred to as deep-brain stimu- 
lation (DBS)—are effective for treatment of motor 
symptoms in PD (5). Furthermore, DBS permits a 
reduction of L-dopa dosage in PD patients (6). 
However, a disadvantage of DBS is the require- 


ment of a highly invasive surgical procedure, as 
well as the crucial dependence on accurate tar- 
geting of very small brain structures (7). Hence, 
it is desirable to identify a less invasive method 
to electrically stimulate neuronal circuits to obtain 
beneficial effects similar to those of DBS. 

Some clues for new PD therapies may come 
from epilepsy studies. In both animal models and 
in epilepsy patients, stimulation of peripheral nerve 
afferents is effective in desynchronizing aberrant 
low-frequency neural oscillatory activity, there- 
by reducing the frequency and duration of seizure 
episodes (8—/0). Aberrant low-frequency neu- 
ral oscillations are well documented in patients 
(/1, 12) and in animal models of PD (/3). These 
findings led us to hypothesize that stimulation of 
afferent somatic pathways could alleviate motor 
symptoms of PD by disrupting aberrant low- 
frequency oscillations. 

Dopamine, akinesia, and synchrony. The first 
set of experiments was carried out using an in- 
ducible mouse model of PD, first in wild-type 
animals and then in dopamine-transporter knock- 
out (DAT-KO) mice (/4). Through pharmacolog- 
ical inhibition of dopamine synthesis, we induced 
acute dopamine depletion in both types of ani- 
mals (2, /3, 14). In patients, the cardinal symp- 
toms of idiopathic PD have been reported to be 
clinically apparent after degeneration of 60 to 70% 
of the dopaminergic neurons of the substantia 
nigra pars compacta, which results in a 30 to 50% 
reduction of striatal dopamine levels (/5, /6). By 
means of two intraperitoneal injections (250 mg/kg) 
of the tyrosine hydroxylase inhibitor alpha-methyl- 
para-tyrosine (AMPT) during a 6-hour period 
(15, 16), we achieved acute pharmacological 
dopamine depletion slightly below the levels 
observed in PD patients in wild-type C57/BL6J 
mice (69% reduction of striatal dopamine levels; 
mean + SD = 4.5 + 2.0 ng dopamine per mg 
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tissue in depleted animals compared with 14.4 + 
3.3 in controls; P < 0.005 Mann-Whitney, 1 = 6/6) 
(fig. S1). Equivalent symptoms of main clinical 
motor manifestations in PD patients were found 
in AMPT-injected mice (figs. S2 and $3). In par- 


Fig. 1. Acute inhibition of dopa- 
mine synthesis produces a Parkin- 
sonian state. (A) Examples of LFP 
spectrograms and firing-rate plots 
recorded in MI during two 5-min 
periods before and after dopamine 
depletion. Top row: Locomotion dur- 
ing recording periods. Second row: 
LFP power. Third row: LFP power 
standardized to the nondepleted 
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ticular, locomotive activity was significantly re- 
duced [average locomotion scores in nondepleted 
and depleted animals were (mean + SEM) 3.7 + 
0.1 and 0.4 + 0.02 mm/s, = 11 and 14, respec- 
tively], and a preferential reduction of faster move- 
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recorded around 300-Hz stimulation events (yellow bar). Top row: LFP power 
(n = 21 events; black trace denotes spectral index). Second row: LFP power 
standardized to first 240 s. Rows 4 and 5, respectively: Firing rate for 98 
striatal and 96 cortical units standardized to firing rates during first 240 s 
and ordered by responsiveness after DCS (19); n = 36 events. Neurons 


21-60 


> 
61-120 121-180 
Locomotion speed (mm/s) 


Locomotion 


Striatal 
neurons 


neurons 


-200 -150 


exhibiting significant changes during the 30-s period after stimulation 
(black line) are indicated with red and blue rectangles (excitatory and in- 
hibitory responses). Middle row: Average locomotion; n = 36 events. 


www.sciencemag.org SCIENCE 


-100 


RESEARCH ARTICLES 


ments indicated bradykinesia in the depleted state 
(fig. S2) (17, 18). 

Neuronal activity patterns of dorsolateral stria- 
tum and primary motor cortex (MI) were also 
significantly altered. Differences were found both 
on the population level, through inspection of 
local field potentials (LFP) and in the firing pat- 
terns of single cortical and striatal neurons (/3). 
Figure 1A shows an example of LFP spectro- 
grams recorded in MI during two 5-min periods 
before and after dopamine depletion (second 
row, left and right, respectively). Spectral anal- 
ysis revealed particularly powerful oscillations 
~1.5 to 4 Hz and in the lower beta range (10 to 
15 Hz), whereas the power of oscillations >25 Hz 
was decreased in relation to baseline conditions 
(standardized spectrograms, Fig. 1A, third and 
fourth row, and fig. S6). 

Important differences in single- and multi-unit 
activity were also found. The firing rates of a ma- 
jority of 52 striatal and cortical neurons, which 
were positively identified after a 6-hour depletion 
period, showed significant differences (70.0% in 
motor cortex and 75.0% in striatum, ao = 0.001) 
when we compared the more active nondepleted 
state and the immobile depleted condition (see 
activity raster plots shown for a few units in Fig. 
1A, bottom row). During dopamine depletion, a 
higher proportion of neurons tended to discharge 
phase-locked to LFP oscillation, in effect result- 
ing in increased synchronicity [52.7% (64/129) in 
depleted versus 37.0% (44/127) in nondepleted 
state; a = 0.001] (see fig. S6 for details). 

DCS alleviates akinesia and synchrony. The 
effect of dorsal column stimulation (DCS) was 
next evaluated in mice before and after acute 
pharmacological dopamine depletion. DCS was 
achieved by chronic implantation of custom- 
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made flat bipolar platinum electrodes positioned 
epidurally above the dorsal columns of the spinal 
cord at the upper thoracic level (Fig. 1, B and C). 

DCS had a dramatic effect on the amount of 
locomotion displayed during stimulation periods 
in the dopamine-depleted animals. This effect was 
strongest for 300-Hz stimulation; on average, 
the amount of locomotion during stimulation pe- 
riods was more than 26 times as high as during 
the 5-min period before stimulation (Fig. 2A and 
Movie S1). DCS had a smaller, albeit clear, effect, 
using lower stimulation frequencies. In contrast, 
control experiments using air puffs or trigeminal 
nerve stimulation were not effective (Fig. 2A and 
fig. S5). DCS caused increased locomotion also 
during nondepleted conditions, but this increase was 
moderate (4.9 times prestimulus values at 300 Hz) 
in comparison to that in depleted animals (Fig. 2A. 
Locomotion was normally initiated a few seconds 
after the onset of DCS, with a slightly longer delay 
in depleted animals (median = 3.35/1.35 s, interquar- 
tile range = 2.22/1.22 s, P=0.023, Mann-Whitey, in 
depleted/nondepleted animals at 300 Hz). In ad- 
dition, a small residual effect was found after 
high-frequency stimulation in depleted, but not in 
nondepleted, animals (3.4 and 0.95 times prestim- 
ulus values, respectively for the 30 s after 300-Hz 
DCS). DCS also proved efficient for alleviation 
of bradykinesia as indicated by the relatively larger 
increase in the amount of fast-movement com- 
ponents in depleted animals (Fig. 2B). 

Analysis of LFP recordings during DCS in 
both MI and in striatum showed a shift in spectral 
power from lower to higher frequencies (average 
spectrograms from a total of 21 events of DCS at 
300 Hz obtained from nine animals are shown in 
Fig. 2C). The spectral shift was maintained through- 
out the stimulation period and lasted for ~5O s 
after the end of stimulation. To condense the spec- 
tral shift into a single measure, a spectral index 
was computed by dividing the spectral range ana- 
lyzed into an upper and lower half and calculating 
the ratio of the summed power of the frequencies 
in the two intervals [(25 to 55 Hz)/(1.5 to 25 Hz)]. 
The spectral index (black trace in Fig. 2C) illus- 
trates the rapid spectral shift induced by DCS and 
the prolonged effect after DCS had ceased. 

DCS also affected the firing patterns of indi- 
vidual neurons. To avoid interference from stim- 
ulation artifacts, the 30-s stimulation periods were 
excluded from the analysis of spike data. Even 
during the period following stimulation, though, 
numerous neurons showed significantly altered 
firing rates (47.9% in MI and 41.8% in striatum, 
a = 0.01; Fig. 2C, rows 4 and 5, respectively). The 
fraction of units entrained to LFP dropped notably 
(from 42.7/38.8% in Ml/striatum the 30 s before 
DCS to 24.5/24.0% the 30 s after DCS, a = 0.01). 

Although the onset of locomotion was delayed 
a few seconds, changes in the neural activity were 
detected almost immediately after DCS onset 
(mean + SD evoked potential latency = 44 + 5 ms), 
perhaps indicating that the electrophysiological 
changes have a permissive rather than a directly 
instructive role for the initiation of locomotion. 


A brain state permissive of locomotion. Dur- 
ing the relatively rare instances when the depleted 
animals displayed locomotion, low-frequency os- 
cillations were diminished (Fig. 1A). This situation 
bears an obvious resemblance to the DCS-induced 
state. Thus, a certain decrease of low-frequency os- 
cillations may be required to initiate locomotion. 
We analyzed the detailed temporal patterns of shifts 


Fig. 3. Activity patterns during spon- A 
taneous locomotion. (A) Average 
spectrogram of striatal LFP aligned 
to the onset of spontaneous loco- 
motion in control (n = 115 events) 
and dopamine-depleted condition 
(n = 51 events). The gradual shift B 
from lower to higher frequencies in- 
dicated by the average spectral index 
(black trace) starts before locomo- 
tion onset (dashed white line). (B) 
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in oscillatory LFP activity during spontaneous loco- 
motion events in nondepleted (115 events in 10 
animals) and depleted (51 events in 5 animals) 
mice (Fig. 3, A and B). In both states, significant 
spectral shifts from lower to higher frequencies, 
assessed by spectral index changes (P < 0.01) (79), 
were detected a number of seconds before the ini- 
tiation of locomotion (nondepleted: mean + SD = 
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non-DCS sessions. In the sham group, in contrast, DCS caused a moderate response comparable to non-DCS 
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2.9 + 1.7 s, range 0.1 to 5.5 s, m = 88, MI and 
striatal LFP; depleted: 3.0 + 1.7, range 0.2 to 5.5 s, 
n = 48, MI and striatal LFP). Yet, there were also 
important differences, most notably below 25 Hz. 
A more differentiated decrease in power of oscil- 
lations below 8 Hz and an increase above 17 Hz 
was observed in nondepleted animals, whereas 
the spectral power in a broader range between 5 
and 25 Hz was decreased in depleted animals. 
Because these different patterns occurred before 
the onset of locomotion, it is unlikely that they 
were due to differences in locomotion between the 
two groups. Instead, they could be part of the ex- 
planation of why depleted animals moved slower 
and for shorter time periods. 

On the single-neuron level, the same type of 
firing rate changes after DCS also occurred in 
conjunction with spontaneous locomotion events. 
From a total pool of 193 neurons (from nine con- 
trol and five dopamine-depleted recording sessions 
in 11 animals), 111 modulated their firing rate dur- 
ing locomotion and, unexpectedly, 59 of these 
neurons showed a patter of early activation, 2.9 + 
1.4 s (mean + SD) before actual locomotion on- 
set (range = 0.5 to 4.5 s, n = 59 striatal and MI 
units from depleted and nondepleted conditions) 
(Fig. 3C). 

DCS in combination with t-dopa treatment. 
To find the minimum dose of L-dopa (alone or 
combined with 300-Hz DCS) required to restore 
locomotion, DAT-KO mice were used. These mice 
have <5% of normal striatal content of dopamine 
(/4). Dopamine can be further decreased to vir- 
tually undetectable levels by injecting AMPT (250 
mg/kg intraperitoneally), resulting in a completely 
akinetic animal model (/4). By gradually increasing 
dopamine levels through repeated L-dopa injec- 
tions every hour, we tested the locomotion thresh- 
olds. In the group receiving only L-dopa injections 
(n = 6 sessions from 4 mice), locomotion typically 
first occurred after the fifth injection (5 mg/kg dose, 
corresponding to a total dose of 15 mg during the 
first 5 hours). When L-dopa treatment was com- 
bined with DCS, the same amount of locomotion 
was displayed after the second injection (2 mg/kg 
dose, corresponding to a total dose of 3 mg in 
the first 2 hours) (1 = 10 experiments from seven 
mice (Fig. 4A). That means that in combination 
with DCS, one-fifth of the L-dopa total dose 
was enough to produce equivalent locomotion to 
L-dopa alone. There was also a general increase 
in the amount of locomotion displayed in the 
L-dopa+DCS group over the entire range studied. 
Thus, L-dopat+tDCS seems to be superior to 
L-dopa alone in terms of the ability to rescue 
locomotive capability after severe dopamine deple- 
tion. Finally, animals in the t-dopat+DCS group 
consistently showed higher values of spectral index 
than the L-dopa only group. This suggests that 
DCS facilitates locomotion, even in severely de- 
pleted animals, through similar mechanisms (fig. S7). 

DCS is effective after chronic lesions. Although 
the acute dopamine depletion model employed 
in the first set of experiments could reproduce all the 
main symptoms of PD, it was important to confirm 


the effectiveness of DCS in an animal model that 
also involves loss of nigrostriatal dopaminergic con- 
nections. Chronic dopaminergic denervation of the 
striatum was achieved using bilateral 6-OHDA le- 
sions in rats (n = 4), resulting in progressive dete- 
rioration of motor function and sustained weight 
loss, both cardinal signs of successful lesioning 
(20, 21). When placed in the open field, lesioned 
rats displayed reduced locomotion compared with 
controls (1 = 4), which received vehicle injections 
in identical sites in the striatum (mean + SEM = 
2.85 + 0.068 and 7.78 + 0.144 mm/s on average, 
respectively). Quantification of immunohistochem- 
ical staining of the dopamine-synthesizing en- 
zyme tyrosine hydroxylase indicated that lesioned 
rats had only ~20% of the immune signal found 
in sham-lesioned animals (fig. S8). 

Rats were tested during two 1-hour sessions in 
the open field, the first hour without stimulation 
and the second with DCS applied for 30 s every 
tenth minute. In the lesioned group, DCS resulted 
in remarkably increased amounts of locomotion 
compared to the first hour, whereas sham animals 
actually moved less during DCS sessions than 
during non-DCS sessions (Fig. 4C). Hence, there 
were specific improvements of motor function in 
the Parkinsonian state compared with controls. In 
lesioned rats, DCS not only alleviated hypoki- 
nesia during stimulation but also caused an in- 
crease in locomotion after the stimulation period. 
This residual effect lasted ~100 s (Fig. 4B). 

The effect of DCS on bradykinesia in 6- 
OHDA-lesioned rats was also evaluated. Le- 
sioned animals showed a relative increase in the 
number of scored locomotion events for all move- 
ment speeds, but this effect was more pronounced 
for faster movements, indicating a specific effect 
on bradykinetic symptoms in addition to the 
general improvement in the overall amount of 
locomotion (Fig. 4D). 

Discussion. We demonstrated that stimulation 
of the dorsal column pathways using epidural 
implanted bipolar electrodes—a simple, easy-to- 
perform, semi-invasive method—can restore loco- 
motive capability in two animal models of PD 
symptoms: acutely dopamine-depleted mice and 
rats with dopaminergic neuronal loss. In parallel 
with the behavioral improvements, DCS shifted 
activity patterns in the primary motor cortex and in 
the dorsolateral striatum into a state closely re- 
sembling that found before and during sponta- 
neous initiation of locomotion in normal and 
depleted animals. This suggests that DCS helps 
motor-related brain areas shift into a state per- 
missive of the initiation of movements. 

What could be the mechanisms through which 
DCS allows a shift into a locomotion-permissive 
state? The first possible explanation could be that 
DCS, in addition to stimulating specific somatosen- 
sory pathways, may also recruit brainstem arousal 
systems, leading to sufficient cortical and striatal 
desynchronization required for voluntary initiation 
of movements (8). Such a possibility can be raised 
when the phenomenon of paradoxical kinesia is 
considered, that is, rare events in which PD pa- 
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tients, aroused by frightening situations, exhibit 
sudden and transient improvement in motor 
function (22, 23). Here, the increase by a factor 
of 4.9 in locomotion produced by DCS in control 
animals, albeit much less than that observed in 
dopamine-depleted animals, could in theory sup- 
port such an arousal hypothesis. Yet, a variety of 
observations suggest that this may not be the 
main mechanism accounting for DCS-induced 
locomotion. First, neither air puffs alone nor stim- 
ulation of trigeminal nerve afferents, both potent 
somatosensory arousal stimuli, induced locomo- 
tion in either control or dopamine-depleted ani- 
mals. Secondly, in control experiments carried out 
in both awake and lightly anesthetized animals, 
DCS produced only a minimal arousal response 
when compared with other tactile, proprioceptive, 
and nociceptive stimuli (fig. S9, A and B). This 
is in line with a previous study that demonstrated 
that dorsal column recruitment produces no signif- 
icant arousal effect (24). Overall, these data suggest 
that DCS may increase locomotion behavior pri- 
marily through direct modulation of lemniscal/ 
thalamic pathways. However, more experiments 
will be required to settle this issue. 

Our electrophysiological data suggest possible 
mechanisms for the success of DCS in restoration 
of locomotion, based on existing theories of basal 
ganglia pathology in PD and specifically consid- 
ering the circuitry known to be involved in initiating 
voluntary locomotion (25). The command to the 
spinal cord to initiate locomotion, via reticulospinal 
pathways, is issued by the diencephalic and mesen- 
cephalic locomotor regions. For these midbrain 
structures to become active and trigger locomotion, 
they must be relieved from the tonic inhibition ex- 
erted by the output nuclei of the basal ganglia. This 
is accomplished by activation of striatal medium 
spiny neurons projecting to the output nuclei of the 
basal ganglia (26, 27). Under normal circumstances, 
the cortex has a powerful excitatory influence on 
the striatum. In contrast, with reduced striatal do- 
pamine levels, the activation threshold of the 
projection neurons from the striatum is significantly 
increased (25), making it less likely that cortical 
input to the striatum will be conveyed through this 
pathway. As a consequence, brainstem motor re- 
gions remain under tonic inhibition, and the initia- 
tion of goal-directed locomotion and other types of 
volitional motor activity become impaired. DCS 
may exert its effect by activating large cortical 
areas, increasing the cortical and thalamic input to 
the striatum. This may, in turn, promote the depo- 
larization and, consequently, facilitate the activation 
of striatal projection neurons. Another consequence 
of the reduced cortical control of striatum at low 
dopamine levels is that both thalamic and internally 
driven striatal low-frequency oscillations become 
more prominent (28, 29). These oscillations may 
lead to increased synchronicity because the gen- 
eration of action potentials tends to occur at 
more distinct phases of the LFP oscillation (73, 30). 
This was confirmed in our experiments in which 
both motor cortex and striatum showed excessive 
low-frequency synchronized oscillatory activity 
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in dopamine-depleted animals and an increased 
entrainment of spikes to low-frequency compo- 
nents of the LFPs. Such synchronous activity in- 
terferes with normal information processing in 
these circuits and should likely be considered 
pathogenic in PD (/2). Our data show that DCS 
effectively abolishes aberrant synchronous low- 
frequency oscillations. It is, therefore, tempting to 
speculate that the suppression of low-frequency 
oscillations is particularly important for ameliora- 
tion of motor symptoms in PD (3/). 

Finally, the combined effect of L-dopa and 
DCS allowed for recovery of motor function at 
significantly lower doses of L-dopa in severely 
dopamine-depleted animals. The considerably less 
invasive nature of the epidural DCS electrode 
compared with DBS electrodes suggests that DCS 
could be a complement for treatment of symptoms 
of PD in earlier stages of the disease. We therefore 
propose that DCS should be investigated further in 
extensive experiments employing primate models 
of PD, preferably over longer time periods, to eval- 
uate the potential viability of this new procedure as 
a treatment for Parkinsonian patients. 
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Alfven Waves in the 
Lower Solar Atmosphere 


David B. Jess,”’2* Mihalis Mathioudakis,? Robert Erdélyi,? Philip J. Crockett,* 


Francis P. Keenan,” Damian J. Christian* 


The flow of energy through the solar atmosphere and the heating of the Sun’s outer regions are 
still not understood. Here, we report the detection of oscillatory phenomena associated with a large 
bright-point group that is 430,000 square kilometers in area and located near the solar disk 
center. Wavelet analysis reveals full-width half-maximum oscillations with periodicities ranging 
from 126 to 700 seconds originating above the bright point and significance levels exceeding 
99%. These oscillations, 2.6 kilometers per second in amplitude, are coupled with chromospheric 
line-of-sight Doppler velocities with an average blue shift of 23 kilometers per second. A lack 

of cospatial intensity oscillations and transversal displacements rules out the presence of 
magneto-acoustic wave modes. The oscillations are a signature of Alfvén waves produced by a 
torsional twist of +22 degrees. A phase shift of 180 degrees across the diameter of the bright point 
suggests that these torsional Alfvén oscillations are induced globally throughout the entire 
brightening. The energy flux associated with this wave mode is sufficient to heat the solar corona. 


olar observations from both ground-based 
and spaceborne facilities show that a wide 
range of magneto-acoustic waves (/, 2) prop- 
agate throughout the solar atmosphere. However, 
the energy they carry to the outer solar atmosphere 
is not sufficient to heat it (3). Alfvén waves (pure 
magnetic waves), which are incompressible and 
can penetrate through the stratified solar atmosphere 
without being reflected (4), are the most promising 


wave mechanism to explain the heating of the 
Sun’s outer regions. 

However, it has been suggested that their pre- 
vious detection in the solar corona (5) and upper 
chromosphere (6) is inconsistent with magneto- 
hydrodynamic (MHD) wave theory (7, 8). These 
observations are best interpreted as a guided-kink 
magneto-acoustic mode, whereby the observa- 
tional signatures are usually swaying, transversal, 


periodic motions of the magnetic flux tubes 
(7, 9). Numerical simulations (/0) show that sub- 
surface acoustic drivers and fast magneto-sonic 
kink waves (//, /2) can convert energy into up- 
wardly propagating Alfvén waves, which are 
emitted from the solar surface. These numerical 
simulations are also in agreement with current 
analytical studies. In particular, it has been shown 
that footpoint motions in an axially symmetric 
system can excite torsional Alfvén waves (/3). 
Other Alfvén wave modes may exist, although 
these are normally coupled to magneto-sonic MHD 
waves (/4). In the solar atmosphere, magnetic 
field lines clump into tight bundles, forming flux 
tubes. Alfvén waves in flux tubes could manifest 
as torsional oscillations (7) that create simulta- 
neous blue and red shifts, leading to the non- 
thermal broadening of any isolated line profile, 
and should thus be observed as full-width half- 
maximum (FWHM) oscillations (75). A promis- 
ing location for the detection of Alfvén waves is 
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Chronic Spinal Cord Electrical 
Stimulation Protects Against 
6-hydroxydopamine Lesions 


Amol P. Yadav', Romulo Fuentes”, Hao Zhang, Thais Vinholo?, Chi-Han Wang’, Marco Aurelio M. Freire? 
& Miguel A. L. Nicolelis'**45 


‘Department of Biomedical Engineering, Duke University, Durham, NC, 27780, 7Edmond and Lily Safra Institute of Neuroscience of 
Natal, Natal, Brazil, 59066-060, *Department of Neurobiology, Duke University, Durham, NC, 27710, “Duke Center for 
Neuroengineering, Duke University, Durham, NC, 27710, °Department of Psychology and Neuroscience, Duke University, 
Durham, NC, 27708. 


Although L-dopa continues to be the gold standard for treating motor symptoms of Parkinson’s disease 
(PD), it presents long-term complications. Deep brain stimulation is effective, but only a small percentage of 
idiopathic PD patients are eligible. Based on results in animal models and a handful of patients, dorsal 
column stimulation (DCS) has been proposed as a potential therapy for PD. To date, the long-term effects of 
DCS in animal models have not been quantified. Here, we report that DCS applied twice a week in rats 
treated with bilateral 6-OHDA striatal infusions led to a significant improvement in symptoms. 
DCS-treated rats exhibited a higher density of dopaminergic innervation in the striatum and higher 
neuronal cell count in the substantia nigra pars compacta compared to a control group. These results suggest 
that DCS has a chronic therapeutical and neuroprotective effect, increasing its potential as a new clinical 
option for treating PD patients. 


Fy arkinson’s disease (PD) is a debilitating neurodegenerative disorder caused by progressive loss of the 

=* dopaminergic neurons of the nigrostriatal pathway’. A variety of pharmacological approaches, of which 
i L-dopa administration is the most effective, have been used to alleviate PD motor symptoms by supple- 
menting the dopamine (DA) deficiency observed in the striatum’. Despite its initial efficacy, long-term admin- 
istration of L-dopa results in fluctuations in the clinical response and in L-dopa-induced dyskinesia (LID), a 
condition which is difficult to treat’. 

The therapeutic approach involving electrical stimulation of subcortical nuclei, known as deep brain stimu- 
lation (DBS), is also very effective in alleviating the motor symptoms of PD°. The best candidates for this 
treatment are idiopathic PD patients with motor fluctuations and L-dopa-induced dyskinesia*. Depending on 
the selection criteria, 1.6 to 4.5% of PD patients are eligible for subthalamic nucleus DBS‘, making DBS available 
only to a small percentage of the overall population of PD patients. 

Based on previous results obtained in three different rodent models of PD°, we have proposed that a new 
neuromodulation procedure that does not invade the brain tissue, known as dorsal column stimulation (DCS), 
has the potential to emerge as an additional therapeutic option for PD patients. In our hands, the most effective 
DCS effects in rodent PD models were obtained when continuous high frequency electrical stimulation was 
delivered to the large superficial fibers, running through the dorsal columns of the spinal cord, by a transversally 
oriented stimulating electrode, implanted at the high thoracic segments of the cord. 

Following the publication of our animal study, a hasty testing of DCS in two patients with advanced PD was 
carried out. This study did not produce any improvement in motor function®. However, these initial negative 
results were easily explained by the significant methodological differences between the stimulation protocols 
employed in the animal and clinical studies. These included: the localization of the electrodes (high cervical in the 
human study vs. high thoracic in the rodent study), and the orientation of the electrode poles relative to the spinal 
cord (longitudinal in the clinical study vs. transversal in the animal experiments). These differences caused the 
effective surface of contact between the electrodes and the spinal cord to be around seven times larger in the 
rodent protocol when compared to the human study. Such a marked difference could account for the activation of 
substantially more ascending dorsal column fibers in the rodents, explaining why such a potent therapeutic effect 
was observed in the animal study and absent in the clinical one’. 
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Soon after, however, support for our hypothesis that DCS can be 
effective as a PD therapy was subsequently obtained with the pub- 
lication of four other clinical reports. These studies clearly indicated 
that DCS, originally intended for treating chronic pain, led to sig- 
nificant alleviation of motor symptoms in a total of 18 PD patients. 
For example, a PD patient, who had been previously implanted with a 
quadripolar spinal cord stimulator at low thoracic level to treat low 
back pain, experienced significant improvement of his motor symp- 
toms during high frequency (130 Hz) DCS*. Furthermore, DCS also 
produced significant improvement of gait, posture, stability, and 
bradykinesia in 15 PD patients who received a spinal cord implant 
designed to treat low back and leg pain’. Next, a patient with 
advanced PD and other sensory symptoms also experienced 
improvement in gait and posture with quadripolar DCS applied at 
thoracic level’®. Lastly, a female patient with PD and chronic neuro- 
pathic pain, experienced increasing improvement of motor PD 
symptoms over a two year period after initiating the DCS treatment. 
This improvement included alleviation of tremor and rigidity, and an 
improvement in gait and posture’’. 

In our view, the pro-kinetic effect of DCS described in animal 
models and in several PD patients could be explained by the activa- 
tion of somatosensory fibers, running through the dorsal column of 
the spinal cord, which led subsequently to the modulation of the 
ongoing activity of somatosensory and motor supraspinal struc- 
tures>’*"’, Accordingly, in our original study, we observed that the 
motor effect of DCS is almost instantaneous and lasts as long as the 
electrical stimulation is maintained (i.e., see the supplementary video 
from previous studies**). 

Up to now, however, the results reported with spinal cord stimu- 
lation in animal models of PD have been limited to its acute effect. 
Nonetheless, DCS is also known to cause changes in gene expression 
at supraspinal structures, which in turn may lead to long-term sus- 
tained effects’*'*. In the present study, therefore, we explored, for the 
first time, the potential effects of chronic delivery of DCS in rats with 
bilateral intrastriatal 6-hydroxydopamine (6-OHDA) lesions. While 
the lesioned rats exhibited sustained weight loss, postural and gait 
abnormalities, paralleled by destruction of dopaminergic striatal 
projections, a group of DCS-treated animals showed a dramatic 
and consistent reversal of these signs, including significant gain of 
body weight gain, marked improvement in motor functions, and less 
dopaminergic neuronal loss throughout the nigrostriatal system. 
These results are compatible with a neuroprotective effect of DCS 
against chemically induced dopaminergic lesions, and suggest that 
DCS could have long-term benefits as a potential new therapy for PD. 


Results 


Chronic DCS prevents severe body weight loss in 6-OHDA lesioned 
rats. Rats received a bilateral 6-OHDA striatal or a sham lesion, and 
their weight and motor symptoms were evaluated for a period of 6 
weeks (See Fig. 1a for time course of the experiment). While the sham 
lesion surgery, conducted on a group of control rats, caused minor 
weight loss that was recovered almost immediately, bilateral 
intrastriatal 6-OHDA lesions resulted in sustained weight loss 
(Fig. 1b). Twice a week DCS treatment (30 minutes per session) 
caused a dramatic recovery of body weight in 6-OHDA lesioned 
rats. Indeed, DCS treated animals not only recovered significantly 
faster than non-treated 6-OHDA rats; their weights were signifi- 
cantly higher from the 11th day post lesion to the end of the 
experiment (Day 11, p < 0.05; days 12-14, p < 0.01; days 15-42, 
p < 0.001, Bonferroni multiple comparisons, DCS treated compared 
to non-treated Fig. 1b). 

Even though the body weight of both 6-OHDA and 6-OHDA + 
DCS rats was significantly lower than that of the sham control ani- 
mals throughout the experiment (ANOVA, two factor experiment 
with repeated measures, groups X days interaction: p < 0.0001), by 


the 6" week, the weight of DCS treated rats approached that of the 
control rats much more than non-treated animals. 

In addition to faster weight recovery, DCS also prevented severe 
weight loss in lesioned rats during the early period after lesioning. 6- 
OHDA + DCS treated rats had a maximum weight loss ([30.94 + 
1.96]% ~9 days post lesion) that was significantly lower than non- 
treated 6-OHDA rats ([40.35 + 1.68]% ~15 days post lesion, p < 
0.05, Mann-Whitney test, Fig. 1c). Weight recovery of 6-OHDA + 
DCS rats began in week 2 and continued to improve until week 6, 
while non-treated 6-OHDA rats started weight recovery only in the 
6th week, implying that chronic DCS almost immediately reversed 
the trend of weight loss. Fig. 1d shows that after initiation of DCS 
treatment, weight change (normalized to week 1) was significantly 
higher at every time point for treated rats as compared to non-treated 
subjects: week 2 (0.68 + 3.37, — 10.38 + 2.08, p < 0.05), week 3 (7.92 
+ 3.95, —8.87 + 3.26, p < 0.05), week 4 (12.26 + 4.52, —8.46 + 2.57, 
p < 0.01), week 5 (21.19 + 3.52, —3.42 + 2.3, p < 0.001) and week 6 
(28.45 + 3.23, 6.32 + 1.66, p < 0.001; all measurements obtained 
with Mann-Whitney test). 

The slope of body weight recovery, calculated from a line connect- 
ing the minimum weight value with the final day value, was also 
higher for DCS treated rats than non-treated rats, (3.15 + 0.21 vs. 
2.2 + 0.33, p < 0.05, Mann-Whitney test, Fig. le) confirming that 
chronic DCS not only prevented sustained weight loss by initiating 
weight recovery earlier, but it also accelerated the process of weight 
recovery in the 6-OHDA lesioned rats. 


Long term DCS restores motor function in PD rats. Bilateral intras- 
triatal lesioning with 6-OHDA resulted in a significant loss of motor 
function. Lesioned rats developed a characteristic crouched posture, 
which we quantified from video images by measuring the length of 
the major axis of an ellipse fitting the animal’s body. The crouched 
posture of the 6-OHDA rats resulted in an axis length shorter than in 
control rats. Quantitative analysis revealed that there was a significant 
interaction between groups and weeks (ANOVA, two factor experi- 
ment with repeated measures, groups X weeks: p < 0.05). Thus, 
starting at week 1.5, a gradual reversal of posture abnormalities was 
observed in the chronic DCS treated rats. Fig. 2a shows that axis 
length was significantly higher for 6-OHDA + DCS rats (n = 6) 
for week 3.5 (63.59 + 0.74 versus 56.54 + 2.10, p < 0.05) and 4.5 (64 
+ 1.79 versus 56.37 + 1.79, p < 0.05) as compared to non-treated 
rats, n = 8 (Bonferroni multiple comparisons). Overall, 30 min 
continuous DCS twice a week was sufficient to restore normal 
posture in 6-OHDA lesioned rats. 

Lesioned rats in this study showed no significant differences in the 
distance traveled during a 30 min open field session between the 
control and the DCS treated animals (Fig. 2b). Nonetheless, control 
rats showed a progressive decrease in traveled distance over the 
weeks, while 6-OHDA rats, both treated and non-treated, showed 
an irregular pattern (ANOVA, two factor experiment with repeated 
measures, groups X weeks interaction: p < 0.0001, but no differences 
were seen on post-hoc analysis). In the same way, the average speed 
displayed in the open field did not show significant differences 
between the groups (Fig. 2c, ANOVA, two factor experiment with 
repeated measures, groups X week interaction: p < 0.01, no differ- 
ences on post-hoc analysis). Yet, the untreated lesioned rats dis- 
played a striking symptom related to locomotion: a rigid gait 
resulting in a non-smooth locomotion (see Supplementary Video). 
This symptom could be quantified by calculating the spectral power 
at the frequency range of 0.5-4.75 Hz of the instant acceleration 
vector of the rat displacement: high power values represent a jerky, 
non-smooth gait (see Supplementary Figure 1 for an example), com- 
patible with rigidity and crouched posture. DCS treated rats, on the 
other hand, showed a much lower spectral power in the same fre- 
quency (0.5-4.75 Hz) of instant acceleration vector. This allowed 
DCS-treated rats to exhibit a smoother gait, suggesting that DCS 


| 4: 3839 | DOI: 10.1038/srep03839 


2 


132 


a A Dorsal column stimulation 
Pres de 6-OHDA A Motor behavioral assessment artiaion 
implant iseton Daily weight measurement for IHC analysis 


ee 


A AR AD) AM) AM) AM AA AA) A AA Ad 


Cc 


—e— Sham control 
—o— 6-OHDA 
*— 6-OHDA + DCS 


= 100 45 * O6-OHDA 
5 ™6-OHDA +DCS 
= 90 o ™ Sham control 
a 35 
oO — 
= 80 30 
F; B 25 
8 8 
= 7 z 20 
oO = 
= © 15 
= 60 us J Es 
3} 10 
a ** RK 
50 5 
0 
40 1 1 1 n 1 1 1 1 n . . 
0 5 10 15 20 25 30 35 #440 Maximum weight loss 
Days post lesion 
d 6-OHDA 
e ™6-OHDA +DCS 
Benne 4 ™ Sham control 
40 H16-OHDA +DCS ae > = 
335 
o 


ed 
mw oO 


% Weight change 
relative to week 1 


Rate of weight gain 
a 


Oo 
ow 


-20 2 3 4 5 6 
Weeks post lesion 


Weight recovery slope 


Figure 1 | DCS improves weight recovery of 6-OHDA lesioned rats. (a) Time course of the experiment. Numbers indicate experimental weeks from 
onset at time 0 (6-OHDA lesion). Rats in6-OHDA + DCS group sustained epidural DCS electrode implantation (week — 1) 1 week before the 6-OHDA 
lesion (week 0). Rats in 6-OHDA and sham control groups underwent only bilateral lesion procedure; 6-OHDA group (total 52.5 tig 6-OHDA), sham 
control group (only vehicle solution). Timing of motor assessment and 30 minute DCS sessions is illustrated by grey and black upward arrows 
respectively. Six weeks post lesion (week 6), brains were collected and processed for immunohistochemistry (IHC). (b) Changes in body weight after 
bilateral intrastriatal 6-OHDA lesion with or without DCS treatment. Lesioned, non-treated rats (n = 8) suffered sustained weight loss with little-to-none 
recovery. Lesioned rats with DCS treatment (n = 7, 30 min, 333 Hz continuous DCS during 30 min twice a week, starting 7th day, black arrow) recovered 
body weight significantly faster than non-treated rats (p < 0.0001, two-way repeated measure ANOVA). *:p < 0.05(day 11), **:p < 0.01(days 12-14), 
** 1 < 0.001(days 15-42), Bonferroni multiple comparisons. c) Maximum weight loss is significantly larger for non-treated rats compared to DCS 
treated rats. *:p < 0.05, Mann-Whitney test. d) DCS treatment reverses the trend of weight loss almost immediately, while non-treated rats continue to 
lose weight till week 5 as shown by weight change relative to week 1, which is significantly higher for treated rats as compared to non-treated. *:p < 0.05, 
**-p < 0.01, ***:p < 0.001, Mann-Whitney test. (e) DCS treatment results in accelerated weight recovery. Rate of weight gain is significantly higher in 
treated rats compared to non-treated. *:p < 0.05, Mann-Whitney test, a.u. (arbitrary unit). All error bars are s.e.m. 


had an effect in improving animal locomotion. Overall, there was a 
significant interaction between groups and weeks in the spectral 
power of the acceleration vector (Fig. 2d, ANOVA, two factor experi- 
ment with repeated measures, groups X weeks: p < 0.05). At week 5, 
lesioned rats (2.46 + 0.66) had higher spectral power than both DCS 
treated (0.91 + 0.17) and control animals (0.96 + 0.13), p < 0.05, 


Bonferroni multiple comparisons. Again, this finding suggests that 
the DCS treatment had a significant effect in improving motor beha- 
vior in lesioned animals. 

Further, the length of the stride was measured at week 4 for ani- 
mals in the three groups. Untreated lesioned rats (58.74 + 2.59) had 
shorter strides as compared to both DCS treated (69.12 + 2.77) and 
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Figure 2 | DCS restores motor functions in PD rats. (a) Changes in rat posture (measured as major axis length, greater length implying better posture) 
with or without DCS treatment. Lesioned rats develop crouched posture resulting in shorter major axis length. DCS treatment restores posture 
significantly faster than non-treated rats [groups X weeks interaction: p < 0.05, two-way repeated measure ANOVA, *:p < 0.05 at week 3.5 and 4.5 
between DCS treated (n = 6) and non-treated rats (n = 8), n.s.: DCS treated rats were not significantly different from controls (n = 4) from week 2, 
Bonferroni multiple comparisons]. (b) Distance travelled and (c) average speed during a 30 min open field session was not significantly different between 
the groups (groups X weeks interaction: p < 0.0001 for distance, groups X weeks interaction: p < 0.01 for speed, but no differences on post-hoc analysis 
for both). (d) Spectral power of the acceleration vector in frequency range 0.5—4.75 Hz (indicating jerky non-smooth locomotion) was significantly 
higher in lesioned rats than DCS treated and controls towards the end [p < 0.05, two-way repeated measure ANOVA, *:p < 0.05 (6-OHDA + DCS 
compared to 6-OHDA), +:p < 0.05 (6-OHDA compared to controls), Bonferroni multiple comparisons]. (e) Stride length measured at week 4 was 
significantly higher for DCS treated rats as compared to non-treated (p < 0.01, one-way ANOVA, *:p < 0.05, **:p < 0.01-Tukey’s multiple comparison 


test). All error bars are s.e.m. 


control animals (79.05 + 4.02), however the stride length of treated 
rats was not significantly different from that of controls (One-way 
ANOVA, p < 0.01, Tukey’s Multiple Comparison test, Fig. 2e). 


Long-term DCS protects nigrostriatal dopaminergic system. 6- 
OHDA lesions resulted in severe damage to nigrostriatal dopaminer- 
gic striatal projections as compared to sham controls, as evidenced by 
the loss of TH immunoreactivity in these areas (Fig. 3a, bottom and 


middle panels). Quantification of the striatal TH immunoreactivity 
by a contrast index (CI) showed significant differences when the 
three groups were compared (one-way ANOVA, p < 0.05, Fig. 3c, 
top panel). 6-OHDA lesioning caused a 67% decrease of striatal TH 
levels when compared to the sham control group (6-OHDA CI 0.114 
+ 0.022, sham lesion 0.34 + 0.024; Bonferroni’s Multiple Compa- 
rison, p < 0.05). However, DCS treatment of the 6-OHDA lesioned 
rats resulted in a 35% decrease of striatal TH levels. The difference 
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Figure 3 | DCS protects nigrostriatal dopaminergic system. (a) Representative immunostaining for tyrosine hydroxylase (TH) in striatum (DAB stain). 
Note the higher dopaminergic innervation in the striatum (CPu, Acb) of DCS treated rat as compared to non-treated, scale bar = 1 mm, CC = Corpus 
Callosum, LV = Lateral Ventricle, CPu = Caudate Putamen, Acb = Nucleus Accumbens. 6-OHDA lesion caused a 67% decrease of striatal TH levels 
(measured by contrast index), with respect to the sham control group, while treatment of the 6-OHDA lesioned rats with DCS resulted only in a 35% 
decrease, (top panel, 3c). The difference between the TH levels of 6-OHDA (n = 6) and 6-OHDA + DCS (n = 6) groups was significant (*:p < 0.05, 
Bonferroni’s Multiple Comparison, (b) Representative immunostaining for tyrosine hydroxylase in substantia nigra pars compacta (SNc), scale bar = 
500 um, SNc = substantia nigra pars compacta, VTA = ventral tegmental area. 6-OHDA lesion resulted in a severe loss of TH immunoreactivity 
(measured by contrast index) in the SNc. 6-OHDA rats showed 74% loss in TH Cl as compared to sham controls while DCS treated rats showed only 44%. 
There was significant difference between the TH levels of 6-OHDA (n = 6) and 6-OHDA + DCS (n = 6) groups in the SNc (*:p < 0.05, Bonferroni’s 
Multiple Comparison, middle panel, 3c. Dopaminergic neuronal cell loss in SNc (expressed as % of neuronal count in sham controls) was significantly 
higher in 6-OHDA (93.32 + 1.62, n = 8) rats as compared to 6-OHDA + DCS (87.43 + 1.95, n = 6) rats (*:p < 0.05, t-test, 2 tailed). All error bars are 
s.e.m. 


between the TH levels of the 6-OHDA and 6-OHDA + DCS groups 
(around 32% of the TH control levels) was significant (6-OHDA + 
DCS CI 0.222 + 0.018, 6-OHDA CI 0.114 + 0.022, Bonferroni’s 
Multiple Comparison, p < 0.05). 

Similarly, 6-OHDA lesioning resulted in a severe loss of TH 
immunoreactivity, measured by the contrast index in the subtantia 
nigra pars compacta (SNc), as shown in the example in Fig. 3b. Six 
weeks after lesioning, 6-OHDA rats showed a significant decrease in 
TH CI (0.106 + 0.014) as compared to sham lesion levels (0.410 + 


0.038; one-way ANOVA followed by Bonferroni’s Multiple Compa- 
rison, p < 0.05), which represents a 74% loss. Meanwhile, 6-OHDA 
rats treated with DCS exhibited only a 44% loss, which was signifi- 
cantly different from non-treated rats (6-OHDA + DCS CI 0.230 + 
0.061, 6-OHDA CI 0.106 + 0.014, p < 0.05, Bonferroni’s Multiple 
Comparison, Fig. 3c, middle panel). 6-OHDA lesioning also resulted 
in severe loss of dopaminergic neurons from the SNc. Neuronal cell 
count depletion (expressed as % of sham control cell count) was 
significantly higher in non-treated rats [(93.32 + 1.62)%, n = 8] as 
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Figure 4 | Neuroprotective effect of DCS on weight and PD symptoms. (a) There was significant correlation between weight and major axis length 
(measure of posture) throughout the experiment (Spearman test, p < 0.0001). (b) Body weight had a significant negative correlation with spectral power 
of acceleration vector (indicating jerky non-smooth locomotion) — Spearman test, p < 0.0001. (c) Spectral power of acceleration vector and speed were 
significantly correlated throughout the experimental period (Spearman test, p < 0.0001). Stride length measured at week 4 had significant correlation 
with major axis length ((d), Spearman test, p < 0.01) and body weight ((e), Spearman test, p < 0.01) indicating that recovery of body weight was 


related to an overall improvement in motor symptoms. 


compared to DCS treated [(87.43 + 1.95)%, n = 6] animals (p < 
0.05, t-test, Fig. 3c, bottom panel). 


Global effects of neuroprotection. We also investigated the poten- 
tial relation between body weight and multiple motor symptoms by 
calculating the linear correlations between them. This analysis 
revealed a positive correlation between weight and major body axis 
length (Spearman test: r = 0.6639, p < 0.0001, Fig. 4a). Likewise, a 
significant negative correlation was found between weight and 


non-smooth gait (Spearman test: r = 0.346, p < 0.0001, Fig. 4b), 
indicating that rats with greater weight loss had severe motor 
symptoms. Body weight also exhibited a significant correlation 
with stride length measured at week 4 (Spearman test: r = 0.7186, 
p < 0.01, Fig. 4e), indicating that an improvement in body weight 
was correlated with an improvement in posture and gait. There was a 
strong correlation between non-smooth gait and speed (Spearman 
test: r = 0.4987, p < 0.0001, Fig. 4c), confirming our claim that the 
non-smooth gait was predominantly observed during high speed 
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rats having all the data variables at a single time point at the end of the experiment [weight (day 42); axis length, acceleration power, distance and speed at 
week 5.5; stride (week 4); TH immunoreactivity of striatum and SNc and SN¢c cell count). Representation of the individual rat data points, using the first 
three PCs on a 3D space, shows that the rats tend to cluster according to their experimental group. Agglomerative cluster analysis of the data (dotted 
ellipsoids) correctly identified the 6-OHDA rats (red dots), but failed to separate the control (blue squares) and the 6-OHDA + DCS (green triangles) 
rats. (b) PCA was performed with four motor parameters [speed, distance, non-smooth gait (acceleration power), and posture (axis length)] for all time 
points and the first two PC scores of all groups were averaged for each time point and plotted against time post lesion. The resulting graph shows the 
progression of the groups throughout the 6 weeks of the experiment. While the three experimental groups start close to each other (week 0.5—1.5), the 6- 
OHDA and 6-OHDA + DCS groups split from control after week 2. By the end of the experimental time, the 6-OHDA + DCS group joins the control 
group (c) Euclidean distances of the average PCs of each group shows that the motor parameters of the 6-OHDA + DCS group started to drift 

from the untreated 6-OHDA group after week 1.5 and became close to control parameters after week 4. 


displacement. A significant correlation between major body axis 
length and stride length (Spearman test: r = 0.6275, p < 0.01 
Fig. 4d) was also observed, indicating that rats with better posture 
had improved gait at the end of the experimental period. Overall, all 
these linear correlations supported our contention that chronic DCS 
treatment improved the clinical effects of 6-OHDA lesions in rats. 

Finally, we performed a cluster analysis using the principal com- 
ponent scores derived from a set of motor variables (posture, non- 
smooth gait, distance and speed at week 5.5, stride at week 4), 
histological variables (TH immunoreactivity of striatum and SNc, 
and SNc cells/slide) and weight at day 42. The cluster analysis cor- 
rectly grouped and isolated the data points from the 6-OHDA rats 
but failed to separate the control and the 6-OHDA + DCS treated 
rats (Fig. 5a). Next, using only the motor variables (speed, distance, 
posture, non-smooth gait), we analyzed how the three groups of 
animals behaved over time, by plotting the progression of the first 
two principal components which account for 79% of the total vari- 
ance. This analysis revealed that, immediately after the lesion, the 
DCS treated group initially clustered together with the 6-OHDA 
lesioned group. Yet, 1.5 weeks afterwards, the DCS treated group 
started to separate from the lesioned group and move towards the 
control group. By the end of the experimental period (week 6), the 6- 
OHDA + DCS group joined the control group’s space and became 
indistinguishable from it (Fig. 5b). This effect can be clearly docu- 
mented by plotting the Euclidean distances of 6-OHDA and 6- 
OHDA + DCS groups from the control group against time 
(Fig. 5.c). At week 1.5 the 6-OHDA + DCS group separates from 
6-OHDA and starts approaching the control’s trajectory, reaching it 
by week 4 (Fig. 5c). 


Discussion 

In this study we quantified for the first time the long-term effects of 
electrical stimulation of the dorsal column of the spinal cord (DCS) 
on body weight, motor symptoms and survival of nigrostriatal dopa- 
minergic neurons in a chronic rat model of Parkinson’s disease. We 
found that chronic DCS applied on a regular basis was associated 
with progressive improvement in characteristic PD motor symptoms 
and accelerated recovery of lost weight. This improvement in clinical 
signs was paralleled with the maintenance of a higher density of 
dopaminergic innervation in the striatum and neuronal cell count 


in the SNc of DCS-treated rats when compared to a group of 
untreated 6-OHDA lesioned animals. These results show that 
long-term DCS is associated with functional and structural recovery 
in a classic animal model of PD, suggesting that this method may be 
considered in the future as a potential therapy for PD patients. 

Comparison between the three groups of animals utilized in this 
study - sham control, 6-OHDA, and 6-OHDA + DCS - revealed a 
progressive weight increase in the control and the chronic DCS 
groups after the initial procedures (sham lesion and 6-OHDA lesion 
respectively). The control group reached its initial weight within 2 
weeks post-surgery, as expected'*'*. Afterwards, this group pre- 
sented a normal weight increase of approximately 3% per week. 
Animals treated with chronic DCS showed a weight increase of 6% 
per week, following the 6-OHDA lesion, which was more prominent 
after 2 weeks of treatment. It is not clear if the increase was only an 
effect of improved motor function (i.e. restoring the ability of the 
animal to feed itself) or increased appetite (due to overall effects of 
treatment) or both. For example, dysphagia caused by oropharyngeal 
dysfunction and hyposmia, which could be responsible for weight 
loss, are common findings in advanced PD patients’’”®. It is also 
suggested that neuroendocrinological dysregulation or lower con- 
centrations of orexins could play an important role in the feeding 
behavior of PD patients*'. Consistent with our present findings, data 
from advanced PD patients subjected to subthalamic (STN) DBS 
shows increased appetite and an average weight gain of 13% within 
~16 months of treatment”, weight gain of 9.7 + 7 kg within 12.7 + 
7.8 months with 60% improvement in UPDRS-III motor scores” and 
a positive correlation between motor symptom improvement and 
weight gain**. Our findings indicate that this latter correlation was 
also obtained when DCS was applied to our PD animals. Future 
studies in our laboratory will address how the activity of neural 
ensembles controlling motor and feeding functions is affected in 
Parkinsonian states. 

The 6-OHDA lesion did not cause quantitative decrease in the 
animal’s average speed and traveled distance. This could be explained 
by the phenomenon known as starvation-induced hyperactivity”, 
which may have masked or compensated for the expected hypoki- 
netic symptoms. Yet the lesioned animals exhibited clear motor 
symptoms, such as crouched posture, short strides, and non-smooth 
displacement across the open field, all of which were significantly 
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reduced by the DCS treatment delivered only twice a week. Using the 
current protocol for deep brain stimulation (DBS) as a benchmark, 
one can postulate that a more frequent treatment or even continuous 
DCS could very likely lead to even larger motor effects. 

At this point it is important to mention what mechanisms could 
account for the motor effects of DCS in PD. Both experimental 
evidence and modeling, obtained to explain the mechanism of 
DCS to treat chronic pain conditions, indicate that electrical stimu- 
lation delivered in the dorsal epidural space, as we did in the current 
work, activates mainly the superficial fibers of the dorsal columns 
and the dorsal roots of the corresponding spinal segment”*. Thus, the 
consensus reached by this literature proposes that the mechanisms 
underlying the DCS effects reported here should emerge from the 
exclusive activation of ascending somatosensory fibers running 
through the dorsal portion of the spinal cord. This activation could, 
in turn, modulate the activity of multiple supraspinal structures, 
including thalamic, striatal and cortical areas®'?. We should, there- 
fore, emphasize that even though higher DCS intensities can addi- 
tionally recruit deeper structures of the dorsal columns, such as the 
corticospinal tract (CST) (which in rodents is located in the ventral 
part of the dorsal columns) we never observed any sign, such as 
muscle twitching of proximal or distal myotomes, that could support 
the thesis that the CST was recruited by our experimental protocol to 
stimulate the spinal cord. Even in humans, where CST are located 
more laterally, DCS at intensities above therapeutic levels can recruit 
fibers of the CST or local spinal motorneurons or interneurons”. 
However, since our experiments were conducted at a stimulation 
intensity that did not cause any muscle twitching, it is highly unlikely 
that activation of CST has contributed in any way to the effects 
described hereby. Yet, given the anatomical differences between 
rodent and primate nervous system (i.e. position of the CST within 
the spinal cord”’) it is crucial to confirm these results in non-human 
primate models. To address that very issue, we have recently con- 
cluded a series of studies showing that DCS produces the same bene- 
ficial effects, in terms of improvement of motor symptoms, in a 
primate model of PD. Such results are currently being prepared for 
publication (personal observation). 

Degeneration of the nigrostriatal dopaminergic system is a com- 
mon finding in postmortem studies of PD patients, and also a good 
indicator of the stage of the disease*®. After long term treatment with 
DCS twice a week, we found that 6-OHDA rats exhibited a moderate 
yet significant reduction in the depletion of striatal TH-staining and 
TH-IR neuronal cells in SNc, as compared to non-treated lesioned 
animals. Although the mechanisms underlying such a reduction in 
dopaminergic degeneration have not been determined, we think they 
might be mediated by increased production or delivery of neuro- 
trophic factors. Previous studies have shown that intrastriatal injec- 
tions of brain derived neurotrophic factor (BDNF) can attenuate the 
effect of 6-OHDA lesions***’. A recent study involving STN-DBS has 
shown proof of a neuroprotective effect on the SNc neurons in a rodent 
6-OHDA model”, while subsequent experiments from the same 
group showed an increase in levels of nigrostriatal BDNF following 
STN DBS™. This could be the case with our DCS treated animals. 
Although clinical studies conducted with advanced PD patients to 
measure disease progression using 18F-fluorodopa PET failed to con- 
firm a neuroprotective effect of clinically effective STN-DBS**, it 
would be interesting to investigate in the future whether neuropro- 
tective effects of chronic DCS can be observed in a clinical population. 

Considering the increased longevity of the population worldwide, 
the introduction of novel treatments that address both the symptoms 
and progressive nature of PD constitute a major priority for the 
management of Parkinsonian patients. Based on recently described 
preliminary evidence showing efficacy of DCS in a series of PD 
patients worldwide*' and our own results in both acute and chronic 
PD animal models, we propose that chronic epidural DCS, a proced- 
ure that does not invade the brain, does not have serious side effects 


and can be carried out at much lower costs and risks for patients, 
could be employed at the early clinical stages of PD to manage some 
of its cardinal motor symptoms. This conclusion is further supported 
by our preliminary data showing that DCS can also alleviate motor 
symptoms in a genetic model of PD in rats and in a primate model of 
PD (R.F. and M.A.L.N. personal observations). 

At this point, there are very few reports of DCS in PD patients. 
Thus, it is difficult to predict if the eligibility of DCS will be substan- 
tially better than for DBS. However, the lack of major side effects, the 
relative ease with which the surgical procedure can be performed, 
and the fact that there is no need to penetrate into brain tissue suggest 
that DCS could become an early stage therapy in the future manage- 
ment of PD patients. 


Methods 


Animals. A total of 41 male Long-Evans rats (body weight ranging from 310 to 450 g) 
were housed individually with ad-libitum food and water in a temperature controlled 
room ona 12 h light/12 h dark cycle. Animal procedures were performed according 
to prior approved protocols by Duke University Institutional Animal Care and Use 
Committee and in accordance with the National Institute of Health Guide for the 
Care and Use of Laboratory Animals (NIH Publications No. 80-23). Rats were 
divided into 3 groups (sham control, 6-OHDA lesion and 6-OHDA lesion + DCS). 


6-OHDA lesion and stimulation electrode implant procedures. Rats in the 6- 
OHDA + DCS group underwent two separate surgical procedures. First, they were 
implanted with spinal stimulation electrodes under anesthesia induced with 5% 
halothane, ketamine (100 mg/kg), xylazine (10 mg/kg) and atropine (0.05 ml). 
Postoperative weight was monitored daily. The implantation procedure was adapted 
from previous studies***. The electrodes were inserted in the epidural space under T2 
(thoracic vertebra) and tied to it with surgical suture. This prevented electrode 
migration and facilitated stimulation over a long period. One week later, after 
recovery of initial weight, rats were anesthetized for a second surgery, with 5% 
halothane, followed by intramuscular injections of ketamine (100 mg/kg), 
xylazine(10 mg/kg) and atropine (0.05 ml). A total of 52.5 ug 6-OHDA 
hydrobromide (Sigma Company, USA - 3.5 mg/ml in 0.05% ascorbate saline) was 
injected bilaterally into the striatum, at 3 locations on each side, using a needle, driven 
by a syringe pump (Sage, Model 361, Firstenberg Machinery Co Inc., USA) via 10 uL 
Hamilton syringe, at 1 uL/min. The needle was left in situ for 5 minutes and 
withdrawn slowly, to prevent backtracking of the drug. Anteroposterior, mediolateral 
and dorsoventral coordinates for the injections were: + 1.0, +/—3.0, —5.0; —0.1, +/ 

3.7, —5.0 and —1.2, +/—4.5, —5.0°’ from bregma. Destruction of noradrenergic 
fibers and terminals was prevented by 1,3-Dimethyl-2-imidazolidinone (DMI, Sigma 
Company, 25 mg/kg), administered IP, 30 minutes prior to 6-OHDA treatment™. 

Rats belonging to the 6-OHDA lesion and sham control groups underwent only the 
surgical procedure required to perform the lesion; animals in the 6-OHDA lesion 
group received bilateral injections of 6-OHDA, while rats in the sham control group 
received only vehicle solution (0.05% ascorbate saline). Time course of the entire 
experiment is shown in Fig. la. The lesion procedures were performed by the same 
individuals throughout all the experiment and all groups were run in parallel. 
Extreme care was taken to consistently maintain the timing of the various methods 
and conditions during the lesion procedure. 


Weight and motor behavioral assessment. In the post-lesion period, rats had access 
to water soaked food pellets and fruit loop cereals and their body weight was recorded 
daily. 15 rats that did not display severe motor symptoms and lost less than 20% of 
initial weight at day 7 post-lesion were discarded from the experiment. Rats with 
severe weight loss (>20% body weight) were retained in their original groups and 
subjected to the entire experiment (N = 18, 10 in 6-OHDA lesion + DCS group, and 
8 in 6-OHDA lesion group). These were labeled as “strongly lesioned rats”. Rats that 
lost more than 25% of initial weight were additionally hand-fed with peanut butter 
daily, until their weight reached 75%. 

A few days after lesioning, strongly lesioned rats exhibited several motor impair- 
ments, including postural and gait instability, and reduced forelimb dexterity’. These 
symptoms manifested into an inability to grasp or chew food, which was clearly 
observed while the rats tried to eat. Motor behavior was assessed twice a week, on days 
4,7, 11, 14, 18, 21, 25, 28, 32, 35, and 39 post lesion. Rats were placed in an elliptical 
open field (85 cm X 70 cm axes, 60 cm tall) for 30 min and motor behavior was 
recorded from a bottom view camera. 

The posture and position of the rats in the open-field was extracted from digitized 
video recordings with custom designed algorithms implemented in Matlab 
(MathWorks, USA). Image processing of single frames from the video was used to 
extract the rat shape. An ellipsoid was fitted to the rat shape, and the length of the 
major axis of the ellipse (expressed in pixels) was used as a measure of posture. Digital 
videos of the ellipse superimposed on rat shape were saved and later used to confirm 
the accuracy of the algorithm. We observed that rats with crouched posture had 
shorter major axis length, while normal rats had longer lengths. For posture analysis 
only those frames during which the rats were mobile were used. Instantaneous speed 
and instantaneous acceleration vectors were calculated from the distance between the 
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X,Y location of the rat every 1/30 seconds in the open field. Lesioned rats showed jerky 
movement that lacked smoothness and fluidity. To quantify this abnormal behavior, 
we calculated the derivative of the instantaneous speed, thus obtaining a measure- 
ment of instantaneous acceleration. A spectrogram with a window of 4 seconds, 
sliding every 0.033 seconds was constructed with the ’mtspecgrame’ function 
(Chronux toolbox) from the acceleration signal. The power spectra between 0.5- 
4.75 Hz of the time bins where locomotion was greater than 5 cm/s were averaged. 
For measuring stride length of rats, videos from top view camera were used. The front 
most part of the camera view was selected for better visualization of hind legs. A 
locomotion bout with at least 3 strides in the selected part of the open field was 
randomly selected from entire session and stride length was measured using ‘implay’ 
function in Matlab and averaged for the 3 strides. Stride was defined as the pixel 
distance between hind leg (right/left) take off point and subsequent landing point of 
same leg. 


Chronic electrical stimulation of the dorsal column of the spinal cord. After the 
30 min behavioral session, rats of the 6-OHDA + DCS group were allowed to rest in 
their home cage for 30 min. Following this period, they were reintroduced in the open 
field and continuous DCS was applied for 30 min. Before each stimulation session, 
stimulation current intensities were determined for each rat, as described in previous 
study’. DCS consisted of biphasic square pulses of 1 ms duration, delivered at 333 Hz 
at a current intensity ~1.2 times the sensory threshold (mean ~ sd, intensity at 
333 Hz was 167 + 52 uA). These intensities ensured that DCS did not cause an 
arousal effect. 


Tyrosine hydroxylase staining and quantification. Forty-two days after lesion, 
animals were perfused with 4% paraformaldehyde and the brains were kept in 30% 
sucrose until sectioning. During tissue sectioning, 30im free-floating sections were 
obtained from the striatum (AP: 2 to —1) and substantia nigra (SN) (AP: —4.2 to 
—6.6), defined according to the rat brain Atlas”. Tyrosine hydroxylase (TH) 
immunohistochemistry was used to study the extent and position of striatal lesions 
and quantify the depletion of dopaminergic neurons in the nigrostriatal pathway as 
described elsewhere"'. 

For quantification of TH-staining, the tissue samples were mounted and pictures 
were taken using a microscope with the same camera configuration and light intensity 
for each slice. TH-reactivity in both striatum and substantia nigra pars compacta 
(SNc) in all groups (sham control, 6-OHDA lesion and 6-OHDA lesion + DCS) was 
assessed by computer densitometry using digital images captured with the camera 
attached to the microscope. Average densitometric values were obtained by using the 
ImageJ software (http://rsb.info.nih.gov/ij/) from 2 images where both structures 
could be unequivocally defined (see Figure 3). The measurements were obtained 
inside a 0.04 mm” square window positioned across the structures of interest. In order 
to evaluate the general TH-reactivity throughout the striatum and SNc we obtained 
three samples per structure. To minimize the effects of within-group variability, we 
adopted a normalized scale based on the non-reactive cortical white matter (averaged 
over measurements of 3 different sites using the same window). For every animal, the 
average optical density (OD) for the striatum or SNc was designated S, for the cortical 
white matter W and a contrast index C was calculated according to the equation: C = 
(S— W)K(S + W)”. 

Using a high-magnification microscope (NIS-element, Nikon, Japan) equipped 
with a software package, a total of 6 SNc slices between AP: 5.8-6.33 mm of bregma 
were used to bilaterally count immunoreactive neurons and an average cell count was 
calculated for every animal. To obtain an unbiased estimate of cell numbers, we 
applied the Abercrombie’s correction factor*, which compensates for the over 
counting of sectioned profiles, using the equation P = A(M/M + L), where P is the 
corrected value, A is the raw density measure, M is the section’s thickness (in 
micrometers) and L is the average diameter of cell bodies (n = 40 by group) along the 
axis perpendicular to the plane of section. The mean value of TH-IR nigral neurons 
for each animal was expressed as a percentage of cell loss, compared to the mean cell 
count for sham control rats. 


Statistical analysis. All results are expressed as mean + sem. Statistical analysis was 
performed using a computer program (Graphpad Prism 5.0, Graphpad Software, 
USA). Weight and motor behavior data was subjected to two-way analysis of variance 
(ANOVA) with repeated measures, followed by post-hoc multiple comparison tests. 
Whenever distributions failed the normality test, non-parametric tests such as Mann- 
Whitney (t-test) were used. Spearman’s Rank Correlation test was used to study the 
correlation between different parameters. 
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SUMMARY 


Although deep brain electrical stimulation can alle- 
viate the motor symptoms of Parkinson disease 
(PD), just a small fraction of patients with PD can 
take advantage of this procedure due to its invasive 
nature. A significantly less invasive method—epidural 
spinal cord stimulation (SCS)—has been suggested 
as an alternative approach for symptomatic treat- 
ment of PD. However, the mechanisms underlying 
motor improvements through SCS are unknown. 
Here, we show that SCS reproducibly alleviates mo- 
tor deficits in a primate model of PD. Simultaneous 
neuronal recordings from multiple structures of the 
cortico-basal ganglia-thalamic loop in parkinsonian 
monkeys revealed abnormal highly synchronized 
neuronal activity within each of these structures and 
excessive functional coupling among them. SCS dis- 
rupted this pathological circuit behavior in a manner 
that mimics the effects caused by pharmacological 
dopamine replacement therapy or deep brain stimu- 
lation. These results suggest that SCS should be 
considered as an additional treatment option for 
patients with PD. 


INTRODUCTION 


Chronic electrical stimulation of subcortical brain structures, a 
procedure known as deep-brain stimulation (DBS), has become 
an important complement to dopamine replacement therapy in 
the symptomatic treatment of Parkinson disease (PD) (Benabid 
et al., 1987). However, partially because of the highly invasive 
nature of this surgical procedure (Morgante et al., 2007) and its 
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need for additional complex and costly technologies, only a small 
fraction of all patients with PD who could possibly benefit from 
this therapy are actually eligible for implantation. In this context, 
the recent demonstration that electrical epidural spinal cord stim- 
ulation (SCS) alleviates akinesia in rodent models of PD (Fuentes 
et al., 2009) and reduces motor symptoms in patients (Agari and 
Date, 2012; Fénelon et al., 2012; Hassan et al., 2013; Landi et al., 
2012) is a significant finding because SCS, unlike DBS, is mini- 
mally invasive. Following the initial report on rodent PD models, 
SCS has been evaluated for treatment of PD in a few clinical 
case studies. Results of these studies range from no measurable 
improvements in two patients (Thevathasan et al., 2010) to signif- 
icant symptomatic relief (Agari and Date, 2012; Fénelon et al., 
2012; Hassan et al., 2013; Landi et al., 2012) in 19 patients. 
More importantly, in some cases SCS achieved PD symptom 
relief equivalent to the best effects obtained with pharmacolog- 
ical treatment (Fénelon et al., 2012). At present, the underlying 
causes for the different outcomes in these preliminary clinical 
studies are not known, but variations in electrode design, spinal 
cord implantation location, and choice of stimulation parameters 
have been suggested as possible contributing factors (Fuentes 
et al., 2010; Nicolelis et al., 2010). One way to optimize the 
application of this potential therapy for PD is to establish what 
neurophysiological changes are associated with the relief of 
symptoms, and to evaluate how these changes can be most 
effectively induced. Here, we have addressed these questions 
by characterizing the behavioral and neurophysiologic SCS 
effects in 6-OHDA (6-hydroxydopamine) lesioned marmoset 
monkeys (Callithrix jacchus), a primate model of PD. 


RESULTS 
Using previously described procedures (Annett et al., 1992; 
Mitchell et al., 1995), five adult male marmosets were caused 


to be parkinsonian through 6-OHDA stereotactic micro-injections 
in the medial forebrain bundle of either one (n = 2) or both 
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hemispheres (n = 3). Injections resulted in neurodegeneration of 
the midbrain dopaminergic neurons projecting to the forebrain 
in the lesioned hemispheres, as assessed postmortem through 
immunohistochemical quantification of tyrosine hydroxylase (Fig- 
ure S1 available online). In lesioned animals, the number of dopa- 
minergic neurons in the midbrain was reduced to 42% + 23% of 
the values observed in control animals. In addition, axonal termi- 
nal staining density in the caudate-putamen decreased to 44% + 
10% of the levels seen in normal subjects. The severity of parkin- 
sonism was regularly examined in all marmosets using manual 
scoring of a wide range of clinical signs/deficits (see Supple- 
mental Experimental Procedures and Movie S1 for details) and 
automated quantification of spontaneous motor behavior. On 
average, spontaneous locomotion was reduced to approximately 
one-fourth of prelesion activity, and PD signs approached the 
maximum score (mean + SD: 75% + 29%) in all the eight cate- 
gories assessed (Bankiewicz et al., 2001; Verhave et al., 2009). 

Once PD clinical signs had been confirmed, animals under- 
went implantation with bipolar epidural SCS electrodes posi- 
tioned symmetrically over the dorsal midline of the spinal cord 
at a high thoracic level (T3-T4). Four of the five animals (two 
with bilateral and two with unilateral medial forebrain bundle le- 
sions) also underwent chronic implantation with microelectrode 
arrays/bundles in both hemispheres for subsequent recording 
of neuronal ensemble activity (both single units and local field 
potentials [LFPs]; Figure S2). These implants targeted multiple 
structures in each animal, including the primary motor cortex, 
putamen, the subthalamic nucleus, globus pallidus pars externa 
(GPe) and interna, and the ventrolateral and ventral posterolat- 
eral thalamic nuclei. In the two unilateral lesion animals, instead 
of the globus pallidus, parts of primary somatosensory cortex 
were implanted. Prior to SCS, all animals were also subjected 
to acute pharmacological inhibition of dopamine synthesis (sub- 
cutaneous injections of alpha-methyl-p-tyrosine [AMPT] 2 x 
240 mg/kg) to further aggravate the PD signs. 

To avoid SCS current intensities that could be experienced as 
uncomfortable, before each stimulation session the intensity of 
each stimulation frequency was adjusted and set to 1.7 times 
the minimum intensity at which any behavioral response could 
be consistently detected (small postural changes, head or neck 
movements). Overall, as previously reported in rodents (Fuentes 
et al., 2009), we observed that SCS induced a clear alleviation 
of motor impairment in monkeys who exhibited severe clinical 
PD signs. Because the stimulation frequencies used in this study 
(range, 4-300 Hz) proved equally effective, the analysis of the 
SCS effects, both behavioral and electrophysiological, were 
performed by pooling all the frequencies, unless otherwise 
stated. All recordings/stimulation sessions were performed in 
freely behaving animals in a transparent acrylic box. Based on 
our automated image analysis of digital videos obtained from 
multiple cameras during neuronal recording sessions, SCS 
induced a 221% increase in general motility of trunk, head, limbs, 
and tail (p < 0.05, Wilcoxon signed rank test, Figure 1A), a 192% 
increase in the frequency of bouts of spontaneous locomotion 
(p < 0.001, two-proportional z test, Figure 1B), and a 144% in- 
crease in the duration of locomotion periods (p < 0.05, Wilcoxon 
rank sum test, Figure 1C). These improvements resulted in a 
243% increase in the total distance covered (p < 0.05, Wilcoxon 


rank sum test, Figure 1D) by the monkeys. Remarkably, SCS 
induced a preferential increase in the fraction of faster locomotion 
components, indicating a specific reduction of bradykinesia, (p < 
0.05, Kolmogorov-Smirnov test of difference in speed histogram 
distributions, Figure 1E). Overall, the resulting distance covered in 
locomotive behavior was practically normalized by SCS (on 
average 91% of intact values, but locomotion differed somewhat 
in that stimulated animals tended to extend bout duration [628%] 
whereas reducing the frequency [11%] compared to the intact 
state). The improvements in motor disability were also assessed 
by an observer blinded to stimulation conditions that rated spe- 
cific clinical signs, such as freezing, hypokinesia, bradykinesia, 
coordination, gait, posture, and gross and fine motor skills, during 
the OFF and ON periods. The motor deficits that exhibited the 
highest reduction during SCS were freezing (31%), hypokinesia 
(23%), posture (23%), and bradykinesia (21%). Overall, the PD 
score showed, on average, a significant reduction of 18.4% + 
13.9% (p < 0.001, Wilcoxon signed rank sum test [including all 
five subjects], Figure 1F; Figure S1). 

In addition to the general clinical improvements observed in all 
monkeys, in a few instances SCS resulted in an extraordinary 
functional recovery. An example of this is shown in Figure 1G, 
where a severely parkinsonian animal, who reached the 
maximum PD score on all clinical signs rated, showed a dramatic 
improvement during SCS. This allowed the animal to find and 
retrieve a food item with no difficulty whatsoever (see Movie S2). 

Chronic, multisite neuronal extracellular activity was analyzed 
both in terms of changes in single neuron’s firing patterns and at 
the level of LFPs. We observed that SCS induced changes in 
neural populations throughout the cortico-basal ganglia- 
thalamic loop in parkinsonian animals (Figure 2A). As shown in 
Figures 2B and 2C, the most noticeable effect was the suppres- 
sion of LFP power in a frequency interval roughly spanning the 
beta-band (8-20 Hz), which was abnormally strong in all PD 
monkeys (respective peak power frequencies for the four sub- 
jects were 10, 11, 12, and 15 Hz; Figure S3A; Stein and Bar- 
Gad, 2013). This suppression of LFP oscillations was observed 
in all animals and, notably, in all parts of the cortico-basal 
ganglia-thalamic loop (although it did not reach significance 
level, p < 0.05, in GPe when comparing averaged power spectra; 
Figures 2B and 2C, p < 0.05, Wilcoxon rank sum test on band po- 
wer 8-20 Hz). In agreement with the behavioral effects, beta sup- 
pression could be obtained using both low- and high-frequency 
SCS paradigms with approximate equal efficacy (average 
LFP power spectra ON/OFF stimulation for all animals and fre- 
quencies are presented in Figure 2B). 

Next, we examined the effects of SCS on single unit activity. 
Overall, approximately one-third of the neurons recorded in the 
lesioned hemispheres displayed significant changes in firing 
rates associated with SCS. In contrast to the effect on LFP and 
motor behavior, SCS modulation of neuronal firing rate differed 
markedly according to the stimulation frequency (Figure 3A; 
see also Figure S3C). Whereas stimulation at low frequencies 
(4-20 Hz) caused mostly excitatory neuronal responses, inhibi- 
tory firing modulation predominated during high-frequency stim- 
ulation (80-300 Hz; Pearson correlation between the ratio of 
inhibitory/excitatory responses and stimulation frequency was: 
primary motor cortex, r = 0.96, p < 0.01; putamen, r = 0.74, 
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Figure 1. Spinal Cord Stimulation Alleviates Motor Symptoms in Parkinsonian Primates 
(A) Average effect on general motility in response to SCS. Each color line represents one recorded animal over all trials. 
(B-D) Average recovery of locomotion: bout distance, bout frequency and duration, respectively (colors represent the four different subjects and asterisks denote 


significant group differences). 


(E) Reduction in bradykinesia reflected by the preferential recovery of faster movement components in locomotion. 
(F) Average improvements in PD score in all testing sessions divided by symptom category (mean and SEM shown). 
(G) Example of functional motor improvement from a state of severe parkinsonism enabling an animal to retrieve food reward using skilled reaching and grasping 


movements. 


p < 0.05; ventral posterolateral thalamic nuclei, r = 0.94, p < 0.01; 
ventrolateral thalamic nuclei, r = 0.76, p < 0.05; subthalamic nu- 
cleus, r= 0.92, p < 0.01; GPe and globus pallidus pars interna not 
significant, Figure 3B). 

Consistent with the LFP oscillatory activity, we observed a 
large fraction of neurons with beta oscillatory firing in the OFF 
period (Figure 3C) that was partially suppressed during the 
SCS ON period (Figures 3D, 3E, and S2). In total, 152 of 273 
(56%) neurons from the lesioned hemispheres displayed signif- 
icant beta range (8-16 Hz) rhythmic firing patterns during the 
OFF periods (p < 0.01, as compared to spectra computed 
from equivalent random spike trains). Of these 152 neurons, 
39 (26%) significantly decreased their beta rhythmicity during 
the SCS ON period. The change in beta power for all units 
with significant rhythmic activity in the beta range is summa- 
rized in Figure 3F. 

Taken altogether, these data suggest that SCS-induced motor 
deficit relief is primarily associated with the disruption of syn- 
chronized oscillatory activity rather than with specific changes 
in firing rate. However, because cortico-basal ganglia activity 
is known to be strongly influenced by behavioral state, neuro- 
physiologic changes could also result, to some extent, from sec- 
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ondary changes in animal motor behavior. Therefore, to further 
clarify how SCS induces neurophysiologic changes that may 
cause symptomatic relief, we compared the neuronal activity 
patterns from the lesioned hemisphere during SCS, in the two 
hemilesioned animals, to either the patterns of the intact hemi- 
sphere or to the lesioned hemisphere following L-DOPA treat- 
ment (subcutaneous 15 mg/kg with benserazide 6.25 mg/kg). 
Recordings were split into 4 s epochs that were classified as 
either active or inactive states based on automatically quantified 
motor activity. To facilitate comparisons between states, two 
separate indices were constructed. First, two vectors were 
created summarizing the mean differences (parkinsonian versus 
intact state) and (parkinsonian versus L-DOPA-treated state), 
respectively, in two multidimensional parameter spaces—spec- 
tral LFP power and firing rate per structure. Each recorded epoch 
could then be represented as a point in the parameter spaces 
and be quantitatively compared to the intact or L-DOPA treated 
state by geometrical projection onto these vectors. Using this 
metric, it was evident that SCS treatment caused a shift toward 
healthy brain activity patterns resembling the effect of L-DOPA 
treatment. This effect was observed only for the analysis of 
LFP spectral power and not for neuronal firing rates (Figure S3B). 
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Figure 2. Spinal Cord Stimulation Alters Neuronal Activity Patterns in Basal Ganglia Circuits 

(A) Example of parallel changes in LFP power in multiple structures of the cortico-basal ganglia-thalamic loop. For each brain structure, right depicts pooled LFP 
spectrograms (brain slice figures reproduced with permission from Palazzi and Bordier, The Marmoset Brain in Stereotaxic Coordinates, Springer Science+- 
Business Media). Note the immediate reduction of low-frequency oscillations (beta band) in response to SCS (red bar, stimulation frequency: 4 Hz; color codes 


denote decibels above pink noise background for LFPs). 


(B) Average LFP spectra for all recording sessions normalized to pink noise showing a significant SCS-induced reduction in LFP beta-power in all structures 


except GPe. Shaded area denotes 95% Cl with 100 bootstraps. 


(C) Changes in normalized firing rates of individual neurons were diverse but, on average, they decreased in response to SCS in globus pallidus pars interna and 


ventrolateral thalamic nuclei. 


Importantly, this shift could not be explained as a secondary ef- 
fect due to an active or inactive behavioral state because a two- 
way ANOVA, used to estimate the relative effect size of SCS 
compared to that of behavioral state, showed that only 2.9% 
and 0.8% of the total variance (eta-squared; using the metric 
for the intact and L-DOPA treated state, respectively) could be 
attributed to behavioral state change, whereas the effect of 
SCS treatment explained 13.4% and 10.8% of the total variance. 

Consequently, the main effect shared by both SCS and L- 
DOPA treatment appears to be the suppression of the excessive 
neuronal population synchronization associated with the parkin- 
sonian state. Although it is not clear how coordinated low-fre- 
quency activity patterns arise in PD, it is possible that an altered 
functional coupling between the circuit elements of the cortico- 


basal ganglia-thalamic loop may be a key underlying factor (Wil- 
liams et al., 2002). To test this idea, we computed the coherence 
of the LFP signals between pairs of different neural structures 
as an indirect measure of their functional connectivity. In the 6- 
OHDA lesioned dopamine-depleted hemispheres, we found 
strong coherence between pairs of structures, but only in the 
parkinsonian low-frequency range (8-15 Hz) (Figure 4A, black 
traces). Importantly, like the L-DOPA treatment, SCS reduced 
the beta coherence (Figure 4A red trace), leading to a significant 
functional decoupling between the different structures. This sug- 
gests that SCS brings the functional connectivity of the cortico- 
basal ganglia-thalamic circuit closer to the normal state of the 
intact brain. Indeed, this decoupling of parkinsonian LFP oscilla- 
tions in the beta band was observed between all the recorded 
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Figure 3. Spinal Cord Stimulation Alters the Firing Rate and Rhythmicity of Neuronal Units in Basal Ganglia Circuits 

(A) Standardized neuronal firing rate response to different SCS frequencies in multiple structures of the basal ganglia circuits (neurons rank ordered according to 
responses). 

(B) The fraction of inhibitory responses increased with higher SCS frequencies. 

(C) Autocorrelograms of two single units in primary motor cortex exemplifying beta-range rhythmic firing pattern in a parkinsonian animal (SCS OFF). 

(D) Autocorrelograms of the same two units showing that the rhythmic spiking is effectively interrupted by SCS. 

(E) The respective power spectra OFF/ON (black/red) for the units shown in (C) and (D). Note the peak (arrow) in the beta-range during the OFF period, which 
disappears during the ON period. 

(F) Changes in power of rhythmic beta-firing plotted for all 183 units that presented significant beta oscillations either in the OFF or ON period. Colored circles 
represent the units with significant suppression in beta power during the ON period. Black line denotes equal power in ON and OFF conditions, thus units located 
to the right of the line display beta suppression. 


structures and was found to be very similar following L-DOPA 2003). Concurrent multisite neuronal recordings showed that 

and SCS (Figure 4B). significant behavioral improvements induced by SCS were 
To further explore the underlying mechanisms whereby SCS _ strongly associated with desynchronization of neuronal activity 

alters network activity, we recorded neural activity while deliv- _ within the cortico-basal ganglia circuitry and reduction in beta- 

ering single or pairs of SCS pulses. These recordings showed __ frequency coherence between structure pairs. We therefore pro- 

that primary somatosensory pathways (ventral posterolateral pose that SCS should be further tested in clinical studies aimed 

thalamic nuclei and primary somatosensory cortex) are activated at measuring its long-term efficacy as a less invasive, long-term 

early by SCS and that a disruption of beta oscillations through a__ therapy for patients with PD. 

phase-reset mechanism appears to cause the observed wide- 

spread desynchronization in the beta band (Figure S4; Fuentes EXPERIMENTAL PROCEDURES 

et al., 2010; Popovych and Tass, 2012). 


Five adult male common marmosets (Callithrix jacchus) 300-550 g were used 
in the study. The animals were housed in pairs in cages (1.0 x 1.0 x 2.3m) ina 
DISCUSSION vivarium with a natural light cycle (12/12 hr) and outdoor temperature. All ani- 
mal procedures were carried out according to approved protocols by AASDAP 
In conclusion, we observed that SCS caused clear clinical Ethics Committee and strictly in accordance with the NIH Guide for the Care 
improvements in a primate model of PD (comparable to, for 24 Use of nee! pate a paisa ale ne. scam iiss 
example, the reported long-term reduction in UPDRS III score aceite ~ 0795-2 Lecce ne ect atime a peeing ag” 
by DBS, ~28%, Follett et al., 2010) and, in particular, for motor Neurotoxic lesions were inflicted with the animals under deep anesthesia. 
signs known to be difficult to treat with DBS. These include def- — Two microliters of 6-OHDA solution (4 mg/ml, 0.05% ascorbic acid, saline) 


icits in posture, gait, and speed of locomotion (Krack et al., were injected into the medial forebrain bundle in (AP/ML/DV): 6.5/1.2/6.0; 
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6.5/1.2/7.0; 6.5/2.2/6.5; 6.5/2.2/7.5; and 6.5/3.2/8.0 (Annett et al., 1992). AP 
coordinates were scaled according to the dimensions of the skull of each 
animal (Stephan et al., 1980). 

The following parkinsonian symptoms were assessed in the transparent 
acrylic box: episodes of freezing, uncoordinated gait, difficulty using fine motor 
skills, episodes of bradykinesia, hypokinesia, balance impairment, and 
posture. The assessment methods were based on previously described pro- 
cedures (Bankiewicz et al., 2001; Campos-Romo et al., 2009; Fahn and Elton, 
1987; Verhave et al., 2009) and are thoroughly described in the Supplemental 
Experimental Procedures. Automatic motion tracking was performed using 
custom developed software in MATLAB. 

LFPs and action potentials were recorded using a multi-channel recording 
system (Plexon). 

Analyses of recorded signals were performed according to previously 
described methods (Fuentes et al., 2009; Halje et al., 2012). 

The position of the recording electrode positions and the extent of dopami- 
nergic lesions were verified through quantitative tyrosine hydroxylase staining 
in all animals. 


SUPPLEMENTAL INFORMATION 


Supplemental Information includes Supplemental Experimental Procedures, 
four figures, and two movies and can be found with this article online at 
http://dx.doi.org/10.1016/j.neuron.2014.08.061. 
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the parkinsonian condition (arrows) that are sup- 
pressed by SCS (red trace; bold line and shaded 
area denote median and _ interquartile range, 
stimulation artifacts around 20 and 40 Hz have 
been removed). 

(B) Connectivity diagram representing the pooled 
LFP coherence in the 8-15 Hz range in relation to 
the 30-40 Hz band between all pairs of electrodes in 
the different structures (values represent averages 
from all five recordings in the two hemilesioned 
animals; all changes in beta-to-gamma coherence 
for SCS ON/OFF are significant p < 0.05, Wilcoxon 
rank sum test). Note that the excessive beta-band 
coherence, represented by the warm colors in the 
parkinsonian state, is effectively reduced by SCS in 
the same way as for L-DOPA treatment. 
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Interface for Epilepsy Control 
Using Dorsal Column Electrical 
Stimulation 
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Amilcar Santos?, Mikhail Lebedev*> & Miguel A. L. Nicolelis’*:>67 


Although electrical neurostimulation has been proposed as an alternative treatment for drug-resistant 
cases of epilepsy, current procedures such as deep brain stimulation, vagus, and trigeminal nerve 
stimulation are effective only ina fraction of the patients. Here we demonstrate a closed loop brain- 
machine interface that delivers electrical stimulation to the dorsal column (DCS) of the spinal cord to 
suppress epileptic seizures. Rats were implanted with cortical recording microelectrodes and spinal 
cord stimulating electrodes, and then injected with pentylenetetrazole to induce seizures. Seizures 
were detected in real time from cortical local field potentials, after which DCS was applied. This 

: method decreased seizure episode frequency by 44% and seizure duration by 38%. We argue that the 

: therapeutic effect of DCS is related to modulation of cortical theta waves, and propose that this closed- 

: loop interface has the potential to become an effective and semi-invasive treatment for refractory 
epilepsy and other neurological disorders. 


Drug-resistant epilepsy constitutes about 22.1% of the total cases of epileptic patients'. Historically, these cases 
have been treated with surgery’, but more recently electrical neurostimulation has emerged as a potential alter- 
native therapeutic approach*. Deep brain‘, vagus”, and trigeminal®’ nerve stimulation, a procedure pioneered in 
our laboratory, have been proposed over the past decade as new alternatives to treat refractory epilepsy. However, 
each of these three alternative therapies has its advantages and disadvantages. For example, deep brain stimula- 
tion (DBS) has a success rate of 60% in patients with refractory epilepsy®, but requires extremely invasive brain 
surgery. Therefore, a smaller number of patients will be eligible for DBS when compared to the other alternative 
therapies’. Trigeminal nerve stimulation (TNS) is far less invasive than DBS, but has a success rate of only 30.2%°. 
Lastly, vagus nerve stimulation (VNS) is also less invasive than DBS, but its success rate is the lowest among all 
three therapies at 24-28% in randomized clinical trials'®!!. 

Electrical stimulation of the posterior funiculus, also known as the dorsal column, of the spinal cord is a 
semi-invasive method!” which we have demonstrated to be effective for Parkinson's disease (PD) treatment in 
rodents!*4 and primates’, and others have shown to be effective in Parkinsonian patients'’*!’. Remarkably, the 
neurophysiological hallmark of Parkinson's disease in animal models is defined by hypersynchronized neuronal 
activity in the beta band of local field potentials (LFPs)'*!°. The LFP patterns observed during these periods of 
hypersynchronized neuronal activity in Parkinson’s disease resembled some of the patterns of hypersynchro- 
nized neuronal activity previously reported in pentylenetetrazol (PTZ) injected rats’*. This latter similarity and 
the fact that this neuronal hypersynchronization can be specifically disrupted by DCS'*-** led us to hypothesize 
that DCS could be used as an alternative treatment for chronic refractory epilepsy. Although a recent study has 
demonstrated that DCS improved seizure related activity in anesthetized rats injected with PTZ”, the full clinical 
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Figure 1. Closed loop brain-machine interface setup. (A) Local Field Potentials recorded from primary 
somatosensory cortex are analyzed in real time. High amplitude signals trigger the microstimulator (Master8) 
which will deliver an electrical pattern to the dorsal columns (DCS). (B) Recording electrodes placement. 

(C) Stimulating electrodes placement (resting in the epidural space between the vertebrae and the spinal cord). 
(D) Raw LFP recording with multiple crossings of pre-established threshold (red dashed lines). The yellow bars 
indicate DCS delivered whenever the threshold was crossed. Bottom: Spectrogram depicting a seizure episode. 


potential of DCS can only be truly addressed in awake animals with DCS being applied in a closed loop mode (ie. 
triggered only when a seizure is detected by an alternative measurement, such as cortical neuronal recordings). 
While PTZ injection may not be the best model to represent the subset of patients with refractory epilepsy”, it has 
provided the most promising results of DCS as an alternative to current neurostimulation techniques’. 

Here we developed a closed-loop brain-machine interface (BMI) that utilized chronic cortical implants to 
detect seizure activity in awake, freely moving PTZ-treated rats (Fig. 1A,B). This BMI also allowed DCS to be 
delivered using the method we previously developed to suppress Parkinson’s symptoms in rodents'*. Overall, we 
observed that this closed-loop BMI substantially reduced the frequency and duration of seizure episodes. 


Results 

A total of 10 rats (six male and four female) were implanted with stimulation and recording electrodes. Several 
days after the animals recovered from this implantation surgery, they were injected with PTZ and the efficacy of 
our closed-loop BMI in suppressing seizure episodes and reducing their duration was examined in 30 experimen- 
tal sessions. Cortical microelectrode implants were placed in the primary somatosensory cortex (S1) and used for 
local field potential recordings (LFPs). Dorsal column stimulation electrodes were placed at the level of vertebral 
T1-T2 segments)!*"4 (Fig. 1C). Two types of experiments were conducted in these 10 animals. 


Experiment 1: BMI-On versus BMI-Off. _ In the first experiment (6 male and 3 female rats; 23 experimen- 
tal sessions), seizure parameters were measured in PTZ-treated rats either with or without DCS driven by the 
closed loop BMI (BMI-On and BMI-Off sessions, respectively). In BMI-On sessions, each time a seizure detec- 
tion threshold was crossed (Fig. 1D), five trains of 200 electrical biphasic pulses (100-200 uAmp) were delivered 
at the frequency of 500 Hz to the dorsal column. In BMI-Off sessions, the recording and stimulation equipment 
were connected the same way, but no DCS was delivered to the animals. 

Injection of PTZ induced characteristic spike and wave discharges (SWDs)*! that were very evident in cortical 
LFP recordings (Fig. 1D) and triggered body twitches as their main behavioral manifestation. SWD frequency 
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Figure 2. DCS improves seizure related activity. (A) DCS reduced the frequency of seizure episodes. The only 
case where the frequency of seizures was not reduced (red line, Fig. 2A), corresponds to a session that ended 
earlier due to technical problems. Symbols X and + correspond each one to a rat with a single BMI-Off or -On 
session. (B) DCS reduced seizure duration. ‘Partial indicates seizures where the BMI was activated only during 
a fraction of the episode. (C,D) Examples of raw LFP signals and corresponding spectrogram for a BMI-Off 
and a BMI-On session. During BMI-Off sessions, pre-ictal activity (approximately 1600 seconds) presented a 
characteristic signature pattern (see text for details). (E) Detail of BMI-Off session presented in C (color code as 
above). (F,G) In BMI-Off sessions, the pre-ictal theta frequency signal was a good predictor of seizure duration, 
however during BMI-On sessions, DCS specifically disrupted this signal. Also, note that long seizures 

(>60 secs) were mostly absent during BMI-On sessions. 


typically increased until a seizure episode occurred (Fig. 1D). Once a seizure was detected, our BMI delivered the 
DCS after each SWD with a 50 ms delay. 

Comparison of the BMI-On and BMI-Off sessions showed that closed loop DCS affected multiple physi- 
ological parameters (Fig. 2A-G). In particular, DCS reduced the overall number of seizure episodes by 44% 
(BMI-On: 0.05 + 0.01 episodes/min; BMI-Off: 0.09 + 0.02 episodes/min; Paired Samples test t= 2.816, df=5; 
P=0.0373; Fig. 2A) as well as the number of SWDs by 72% (BMI-On: 1.8 + 0.3 SWD/min; Off: 6.5 + 2.6 SWD/ 
min; Wilcoxon signed-rank test = 21; P= 0.0313). Additionally, DCS reduced seizure duration by 34.86% 
(BMI-On: 31.39 + 2.4 secs; Off: 48.19 + 3.5 secs; Min: 9 secs; Max: 136 secs; Mann-Whitney U =551.5; P=0.0012, 
Fig. 2B; Partial indicates episodes where the BMI failed to deliver DCS). No differences were found in seizure epi- 
sode characteristics when rats, tested in the same conditions, were compared across consecutive sessions (paired 
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samples t-test; Duration: T=0.3314, df=4; P =0.7570, n.s.; Frequency: T=0.48, df =4; P=0.67, n.s,), suggest- 
ing that the differences between BMI-On and -Off sessions were not due to repeated PTZ administration. 

Further analysis of the distribution of seizure episode durations showed that DCS negatively skewed this 
distribution, meaning that long seizure episodes (longer than 60s) became much less frequent (BMI-On: 
1/30 = 3.3% episodes; BMI-Off: 11/62 = 17.74% episodes; Fisher's exact test: P= 0.048; compare Y axis values in 
Fig. 2 between KG). 

Frequency spectral analysis indicated that DCS specifically disrupted the LFP spectral pattern that preceded 
the onset of each seizure episode” in PTZ treated rats. This LFP pattern consisted of an elevated theta band 
(~4 Hz to 8-10 Hz), which often appeared as a parabola”’. These PTZ-related theta episodes, which usually 
lasted approximately 5-10 s (compare Fig. 2 panels C,D), occurred in a very narrow range of frequencies and 
occasionally appeared in higher harmonic frequencies (see arrow in Fig. 2C, also 2E). Thus, although pre-ictal 
activity very often included other bands, spectrogram changes associated with the period occurring immedi- 
ately before the seizure episode most reliably appeared in the theta frequency. During BMI-On sessions, this 
PTZ-induced elevated theta band pattern was disrupted. This means that, after the delivery of DCS, the specific 
parabola pattern was no longer present even when this frequency band still presented a high potency signal. The 
main effect observed was an increase of LFP power in a wide theta range (4.5-8 Hz) (BMI-On: —27.57 + 1.4 dB; 
BMI-Off: —33.01 + 1.3 dB; t=2.64, df= 90; P = 0.0098; also see right shift in X axis values in Fig. 2E,G). Lastly, 
DCS also induced longer periods with reduced pre-ictal theta band power (BMI-On: 2.81 + 1.81 secs; BMI-Off: 
1.51 +0.13 secs; Mann-Whitney U =306; P = 0.0038). Thus, DCS induced a reduction in the proportion of long 
seizure episodes, an increase in theta power and range (compare Long and Regular in Fig. 2K,G), and allowed for 
longer periods with low power in the theta band. 

These findings suggest to us that the elimination of the theta pattern by DCS may have accounted for the 
mechanism that led to seizure reduction. In support of this theory, we observed that theta band power dur- 
ing the pre-ictal period was a good predictor of longer seizure duration in BMI-Off sessions (F, 5; = 17.09; 
R?=0.23;P < 0.0001: see Fig. 2E,F). By contrast, during BMI-On sessions, theta band power was no longer corre- 
lated to seizure duration (BMI-On: F 7x, = 0.32; R?=0.01;P = 0.579, n.s.; Fig. 2G). 


Experiment 2: Mixed BMI on and off episodes within asession. To test how fast our BMI became effec- 
tive in reducing PTZ induced seizures, we turned the BMI on and off periodically within the same experimental 
session. We called these experiments the mixed sessions (N =7 rats, 4 male and 3 female in seven sessions; see 
Fig. 3A). Seizure episode durations now varied between 9 and 76 seconds. Under these conditions, we found that 
our closed-loop BMI still drastically reduced episode duration by 42.15% (BMI-On: 26.5 + 2.1 secs; BMI-Off: 
45.81 + 3.2 secs; Mann-Whitney U=74; P < 0.0001; see Fig. 3B; Partial indicates episodes where the BMI failed 
to deliver DCS). 

As in the case of the first experiment, pre-ictal theta band power was a good predictor of seizure duration 
in the mixed sessions when the BMI was off (F, }; = 5.80; R?=0.28; P = 0.0293; Fig. 3C). Once again, when the 
BMI was on, the theta band power no longer correlated with seizure duration (BMI-On: F, »,= 0.38; R’=0.01; 
P=0.54, n.s.). Conspicuously, analysis of long seizure episodes (i.e. >60 seconds) now revealed that turning the 
BMI on in a fraction of the seizure episodes significantly reduced the number of these long seizures even when 
the BMI was off (BMI-On: 0/30 = 0% episodes; BMI-Off: 2/17 = 11.8%; Fisher’s exact test: P= 0.145, n.s.; also see 
Fig. 3C, compare Regular to Long). This finding suggested that, during the course of a PTZ session, the delivery 
of the DCS pattern during one seizure episode could, to some extent, affect the characteristics of the following 
episode!*-15, even if no DCS was delivered at that particular episode. In other words, we found evidence for a 
long-lasting effect of DCS, similar to what we had reported before when we used DCS to treat rat and monkey 
models of Parkinson's disease!*-"». 

To further test this possibility, we looked at the characteristics of the pre-ictal theta band signal, which in 
Experiment 1 was very different between BMI-On and BMI-Off sessions, during the mixed sessions. In this 
latter case we found that, not only was the pre-ictal theta band signal potency now similar between BMI-On and 
BMI-Off seizure episodes (BMI-On: 30.50 + 1.5 dB; BMI-Off:27.25 + 2.2 dB; paired samples t test t= 1.452,df=6; 
P=0.1968, n.s.), but that the theta band amplitude signals obtained during the BMI-Off episodes were now 
closer to those measured during BMI-On episodes (also compare values in X axis in Fig. 2F to values in Fig. 3C). 
Lastly, analysis of low power theta band durations (which in experiment 1 were smaller for BMI Off episodes), 
also revealed that these were now similar between BMI-On and BMI-Off episodes (BMI-On: 2.31 + 0.34 secs; 
BMI-Off: 2.44 + 0.38 secs; Man-Whitney U = 163.5, P =0.7084, n.s.). Therefore, these results suggest that, in this 
experiment, BMI-Off episodes where, to some extent, affected by DCS delivered during the BMI-On episodes. 


DCS is effective in both male and female rats. To identify possible gender specific differences in our 
results”, we further pooled male or female rats from both experiments and compared the main findings of this 
study according to animal gender. The use of DCS reduced the overall duration of seizure episodes in both male 
(t-test with Welch’s correction, t = 4.665, df =59, P < 0.0001) and female rats (t-test with Welch’s correction, 
t= 3.563, df=59, P=0.0007). Additionally, the pre-ictal theta band signal was predictive of seizure episodes 
in both male (BMI-Off: Fy 4. = 13.37; R’ =0.24;P = 0.0007) and female rats (BMI-Off: F, 5; = 4.357; R?=0.12; 
P=0.0447) when DCS was Off, but not when it was On (BMI-On Male: Fj 55 = 0.42; R?=0.01;P = 0.52, n.s.; 
BMI-On Female: F; »3= 0.30; R?=0.01;P =0.59, n.s.). 


Discussion 

We demonstrated here the efficacy of a closed-loop BMI that triggered DCS in response to pre-seizure and sei- 
zure patterns in LFP activity in PTZ-treated rats. Overall, we observed that our BMI quite effectively reduced the 
number of seizure episodes, and their duration, while also changing the overall pattern of LFP activity associated 
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Figure 3. Intermittent delivery of DCS improves seizure related activity. (A) Example of session where the 
BMI was turned On or Off in successive seizure episodes. (B) DCS reduced seizure duration. Partial indicates 
seizure episodes where DCS was delivered only in a fraction of the episode. (C) In BMI-Off episodes, the pre- 
ictal theta frequency signal was a good predictor of seizure duration. During BMI-On episodes, DCS specifically 
disrupted this signal. Also, note that long seizures (>60 secs) were absent during BMI-On episodes and during 
BMI-Off episodes as well (see text for details). 


with the pre-ictal phase of PTZ-triggered seizures. Therefore, we propose that the main anti-seizure effect of DCS 
is obtained via the reduction in the pre-ictal theta band activity, a good predictor of seizure duration. Lastly, we 
found that our BMI was effective in both male and female rats, even though our experiments were not controlled 
for the estrous cycle”. 

Previous studies have shown increases as well as decreases in epileptic related activity after treatment with 
DCS'*4, Here, we have specifically used DCS in response to a change in LFPs signal and consistently observed 
improvement in multiple physiological parameters. We attribute the differences between our findings and previ- 
ous results to the fact that we delivered DCS only in response to LFP changes instead of stimulating indiscrimi- 
nately4. Another important factor is that we only employed high frequency DCS in the present study, since we 
and others have observed (D.G.: personal observation) increased seizure activity when low frequency DCS was 
delivered”. Using transcranial electrical stimulation in a different model, Berenyi et al. have developed a closed 
loop BMI for epilepsy”!. While that study was able to achieve reductions in seizure related activity somewhat 
higher than the ones achieved here, it is important to note that they used a different chemical agent. Future studies 
comparing different BMI approaches and epilepsy models will help identifying pros and cons, as well as efficacy, 
of each technique. At this point it is important to recall that the PTZ model - as used here - may not be the best 
animal model to represent the subset of patients with refractory epilepsy”®. Therefore, the effects of our closed 
loop BMI will have to be further tested in other animal models of epilepsy. 

It could be argued that the differences found between BMI-On and -Off seizure episodes reported here could 
be the result of differences originating from repeated PTZ administration. Although repeated administration of 
PTZ is often used as a model for chronic seizures (see Erke¢ and Arihan 2015 for a review)” our results cannot 
be explained by such effect alone. First, in experiment 1, not only BMI-On and -Off sessions were typically alter- 
nated, but some rats started with BMI-Off sessions while others started with BMI-On sessions. Second, there was 
no difference in seizure episodes (duration and frequency) in rats tested in the same conditions in consecutive 
sessions. Third, most PTZ kindling protocols involve more than 10 doses of PTZ” or intervals of more than 
20 days between the series of PTZ injections”*. Lastly, results from experiment 2 (where the BMI was turned On 
and Off within the same session) further controlled for the possibility of differences in BMI-On and -Off sessions 
being the result of PTZ-induced kindling alone. 

This study also reports, for the first time, that the pre-ictal theta band signal can be used as good predictor 
for seizure episode duration in PTZ-treated rats. It is not yet clear why this signal is related to the duration of the 
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seizure episodes, but a possible explanation involves a mechanism where a state of seizure derives from an abnor- 
mal transition between brain states resulting from an imbalance between corticofugal inhibition and thalam- 
ocortical excitation. In healthy rodents, strong increases in theta band signal, with partial increases in other 
frequencies, are also present during whisker twitching”**°, a non-pathological seizure-like state controlled by S1 
that is characterized by general immobility coupled with improved ability to detect incoming tactile stimuli. From 
this cortically controlled state, neural activity does not usually evolve to seizure episodes, but rather transitions to 
a state of quiet waking where the animal is either immobile or engaged in stereotyped behaviors*’. 

Ina rat model of cortical injury generated epilepsy, an initial stimulus from the injured area to the thalamus 
will make the thalamocortical loop transition to a hypersynchronization state characterized by seizures*. This 
state is maintained by the thalamus and can be reversed by optogenetic thalamic stimulation*. Together, these 
findings suggest that many of the differences found in theta power in this and previous studies could be the result 
of this critical balance in the thalamocortical loop where S1 corticofugal inhibition** maintains theta oscillations 
within a normal range (i.e. whisker twitching), but that otherwise, if theta oscillations become mostly dependent 
on a hyper excitable thalamus**™, this state will then transition to a state of hypersynchronized seizure activity. 
Note that such a mechanism could additionally explain the differences found in previous DCS studies. Thus, if 
theta band activity critically reflects a balance between thalamic and cortical activity, electrical stimulation to 
the lemniscal pathway could result in thalamic increased excitability or, if sufficiently strong, it could in addition 
stimulate $1 and increase cortico-thalamic inhibition*’. The differences in seizure related activity found when 
DCS was applied with low or high frequencies can be partially explained within this framework. Low frequency 
DCS would increase seizure activity because it should affect mostly thalamocortical synapses, while increasing 
stimulation frequency should be able to increase both the thalamus and S1 (Supplementary Figure S1), therefore 
activating the corticofugal synapses and improving seizure related activity (ref. 19 and here). Lastly, this critical 
balance between S1 inhibition and thalamic excitation could also explain the predictive power of the pre-ictal 
signal. The observation that S1 controls the state of whisker twitching (which is characterized by high potency 
theta oscillations) suggests that the bimodal distribution of the predictive pre-ictal theta signal found here may 
actually correspond to two different brain states resulting from the initial conditions imposed by the pre-ictal 
theta signal (one prone to long seizures and another one prone to short seizures). In this scenario, a theta signal 
smaller than —45 dB in S1 (Fig. 2F) after the injection of PTZ would constitute the critical potency required to 
disrupt the balance in the thalamocortical loop, transitioning to a state of long seizures, while a theta signal larger 
than —45 dB, while still disrupting the balance in the thalamocortical loop, would promote transition to a state 
where short seizures occur. It is important to note however that DCS activates multiple regions, making it unlikely 
that the proposed mechanism would be the only source for the brain state transitions described. Future studies 
involving recordings and stimulation across the thalamocortical loop will allow dissecting to what extent DCS 
affects this theta signal in each structure as well as its significance in different thalamocortical states. 

Electrical neural stimulation has been used as an alternative to surgery for intractable epilepsy cases, primarily 
through deep brain stimulation, vagal nerve stimulation, and trigeminal nerve stimulation. Deep brain stimu- 
lation has presented an efficacy of up to 68% responders after 5 years** but is extremely invasive and cannot be 
performed in many patients. Meanwhile, vagal and trigeminal nerve stimulation procedures, have demonstrated 
relatively low efficacy, with 49%*° and 50%*’ respectively, and may have more side effects** than deep brain stim- 
ulation. Thus, on one side, deep brain stimulation has achieved very good seizure reduction, but is extremely 
invasive and expensive, making it a solution for only a small fraction of the patients in need. On the other side, 
more peripheral stimulation procedures, which are much less expensive and invasive, have a much lower efficacy 
rate and are associated with increased side effects. Finally, DCS seems to rest in the middle ground between these 
other electrical neurostimulation alternatives, since it’s less invasive than deep brain stimulation and few side 
effects have been reported when DCS is used for other diseases*’. Translating our findings into human patients 
will allow comparison of the efficacy rate between DCS and these other alternatives. 

Previously we have shown that DCS ameliorates symptoms of Parkinson's disease by desynchronizing patho- 
logical low frequency corticostriatal oscillations, therefore creating a brain state permissible for the initiation 
of locomotion in severely dopamine depleted rodents and non-human primates!*'5. More specifically, high 
frequency DCS inhibited oscillatory neuronal activity synchronized at the beta frequency in the basal ganglia 
through activation of various structures along the dorsal column medial lemniscal pathway’’. Our present PTZ 
results are in line with these previous Parkinson's disease studies, by suggesting that DCS is responsible for the 
desynchronization of pathological synchronous activity characteristic of the PTZ model of epilepsy. In fact, this 
observation can be generalized to other disorders, since we and others have demonstrated that pathological 
synchronous activity seems to be the hallmark of pathological brain states recorded from multiple models of 
neurological and neuropsychiatric disorders such as Parkinson's disease!>-', epilepsy (here and ref. 19), bipolar 
disorder”, and schizophrenia“’. Therefore, based on this cumulative body of evidence showing abnormal timing 
in brain circuitry, we propose that the aforementioned diseases can all be classified within a broad spectrum of 
pathological timing brain states (i.e. hyper- or hyposynchronized), that resemble the basic neurophysiological 
hallmarks of epilepsy. While these diseases share excessive synchronization as a common feature, they differ in 
the type of neural circuits involved in each case”. A testable prediction of this hypothesis would be that any type 
of nerve stimulation capable of significantly altering the balance between regions responsible for these synchro- 
nizations, should also be able to induce at least a partial relief of symptoms in these disorders”. 

In conclusion, we propose that DCS should be tested in other rodent and primate models of chronic epilepsy 
to measure its efficacy in controlling these pathological brain states over longer periods of time. These studies 
would be essential to determine the true potential of DCS as a non-pharmacologic alternative therapy for use in 
humans suffering from chronic, untreatable epilepsy. 
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Methods 

All animal procedures were performed in accordance with the National Research Council’s Guide for the Care 
and Use of Laboratory Animals and were approved by the Duke University Institutional Animal Care and Use 
Committee. Long Evans male and female rats weighing between 250-400 g were used in all experiments. 


Surgery for microelectrode array implantation. Animals went through two different surgeries: one to 
implant recording electrodes and the other to implant the stimulation electrodes. Recording electrodes: Fixed or 
movable microelectrode bundles or arrays of electrodes were implanted in the S1 of rats and additional regions 
(for the present study we did not evaluate the activity in other regions). Anesthesia was induced with 5% halo- 
thane, and maintained with ketamine (100 mg/kg), xylazine (10 mg/kg) and atropine (0.05 ml). Craniotomies 
for S1 recordings were made and arrays lowered at the following stereotaxic coordinates: [(AP) —3.5mm, (ML), 
—5.5mm (DV) —1.5mm]“*. Stimulation electrodes for spinal stimulation were also implanted under anesthesia 
as described above. Postoperative weight was monitored daily. The implantation procedure was performed as 
previously described". Specifically, stimulation electrodes were inserted in the epidural space under thoracic 
vertebra T2 and, to prevent electrode migration, were tied to it with surgical suture. 


Electrophysiological recordings. A Multineuronal Acquisition Processor (64 channels, Plexon Inc, 
Dallas, TX) was used to record neuronal spikes, as previously described“. Briefly, neural signals were recorded 
differentially, amplified (20,000-32,000X), filtered (filtering band between 400 Hz and 5kHz), and digitized at 
40 kHz. Local field potentials (LFPs) were acquired by band-pass filtering the raw signal (0.3-400.0 Hz), pream- 
plified (1,000), and digitized at 1,000 Hz using a digital acquisition card (National Instruments, Austin, TX) and 
a multineuronal acquisition processor (Plexon). 


Pentylenetetrazole administration. Each recording session, independently of the experiment, was per- 
formed on a different day. Both male and female rats were tested under the exact same conditions. PTZ (SIGMA 
Aldrich) administration was prepared by dilution of 100 mg/kg of PTZ in 1 ml saline. This was then administered 
IP under isofluorane anesthesia. As BMI sessions were preceded by an initial baseline recording period, rats 
injected with the PTZ could be immediately brought to the recording room with a delay of no more than 5 min- 
utes. BMI sessions started approximately 5-10 minutes after the administration of PTZ. The recording sessions 
(in both experiments) lasted 60 minutes. In preliminary experiments we observed that PTZ effects were less var- 
iable within the first 60-90 minutes. 


Data analysis. Neuronal data obtained from a total of 30 recording sessions was processed and analyzed 
using NeuroExplorer (version 3.266; NEX Technologies, Madison, AL) and custom scripts written in Matlab 
(12.0; Mathworks, Natick, MA). A seizure episode was defined as a period where observable muscle spasms and 
high amplitude oscillations in raw LFP trace, were accompanied by increased power across multiple frequency 
bands. Seizure episodes were initially identified during the session using both behavior and raw LFP traces as 
indicators, and later confirmed through detailed reanalysis of raw LFP traces and spectrograms. Comparison 
of seizure and SWD frequencies (calculated in seizure episodes or SWD events per minute) was made using a 
paired samples t test or Wilcoxon signed ranks test. When an animal had more than one BMI-On or BMI-Off 
session, a single value resulting from the mean of the sessions was used for comparison. Seizure duration was 
compared using the Mann-Whitney test. Analysis of the overall distribution of seizure durations indicated a 
bimodal distribution. Accordingly, seizure episodes were analyzed as Long (>60 secs) or Regular (<60secs). Then 
the proportion of Regular and Long seizure episodes was calculated for BMI-On and BMI-Off episodes. Lastly, 
the proportion of Regular and Long seizure episodes was compared using Fisher’s exact test. These calculations 
were performed separately for each experiment. For comparison of pre-ictal theta band spectrogram power we 
used values from 4.5-8 Hz frequencies in the 5 seconds before the timestamp that was identified as the start of 
the seizure episode. Theta power was calculated from the original signal processed in Neuroexplorer, followed by 
processing with custom scripts written in Matlab. Values were normalized with the Log of power spectral density 
(dB) and initially analyzed in bins of 100 ms. Calculation of theta power for correlation was made using a single 
5 second bin (the 5 seconds immediately before seizure onset) for theta frequency that was then correlated to sei- 
zure episode duration. For ease of presentation, spectrograms are presented in bins of 100 ms and smoothed with 
a Gaussian filter of 300 ms. Statistical comparison of pre-ictal theta band power was made using an independent 
(Experiment 1) or paired (Experiment 2) samples t test. Pearson correlation was calculated using seizure episode 
duration and the pre-ictal theta power. The duration of theta band potency decrease was compared using data 
from the spectrogram of the whole session initially processed in 1 second bins in Neuroexplorer. A Zscore was 
calculated for theta band frequency for each bin across the session, and then periods of 5 seconds before the onset 
of each seizure episode were analyzed. As increased theta band Zscores were present almost exclusively during 
seizure episodes or spike-and-wave discharges, we analyzed instead periods where Zscores decreased (i.e. indicat- 
ing a low potency theta band signal). Decreases in theta potency were considered here as a Zscore equal or below 
1.0 standard deviation. The duration of each response was then considered as the number of consecutive bins 
where the potency of the signal corresponded to this criterion. Lastly, duration of low theta power responses was 
compared between BMI-On and BMI-Off episodes in each separate experiment, using the Mann-Whitney U test. 


Brain-machine interface based on Dorsal Column Stimulation. Our brain-machine interface used 
Dorsal Column Stimulation (DCS) cues that were generated by an electrical microstimulator (Master 8. AMPI, 
Jerusalem, Israel) controlled by a custom Matlab script (Natick, USA) receiving information from a Plexon sys- 
tem over the internet. This real-time neural analysis and stimulation system has been previously described for a 
different purpose*®*”. Here, we have pre-determined for each rat, a threshold in raw LFP traces that was typically 
crossed only in the presence of LFP epileptic activity (i.e. spike and wave discharges or seizure episodes). Upon 
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detection of such threshold crossing, a pattern of 200 (bipolar, biphasic, charge balanced; 200 sec) pulses at 
500 Hz was delivered to the dorsal column of the spinal cord at the level of T1-T2 segments. Current intensity var- 
ied from 100-200 1A. Seizure episodes where DCS failed to stimulate for at least 75% of the episode duration were 
considered as ‘Partial’ stimulation and were excluded from final analysis. These included a total of 6/53 = 11.32% 
episodes in Experiments 1 and 2. 


In Experiment 1 rats were typically tested in BMI-On and BMI-Off sessions on alternate days. Similarly, in 


Experiment 2, rats seizure episodes with BMI-On were alternated with BMI-Off episodes. Changes to these 
pre-established conditions were made when technical problems occurred (e.g. incomplete session, cable discon- 
necting, noise, inadequate threshold etc.). 


References 


1. 


Picot, M. C., Baldy-Moulinier, M., Daures, J. P., Dujols, P. & Crespel, A. The prevalence of epilepsy and pharmacoresistant epilepsy 
in adults: a population-based study in a Western European country. Epilepsia 49, 1230-1238, doi: 10.1111/j.1528-1167.2008.01579.x 
(2008). 


. Horsley, V. B-surgery. Br. Med. J. 2, 670-675 (1886). 
. Mogul, D. J. & van Drongelen, W. Electrical control of epilepsy. Annu. Rev. Biomed. Eng. 16, 483-504, doi: 10.1146/annurev- 


bioeng-071813-104720 (2014). 


. Fisher, R. et al. Electrical stimulation of the anterior nucleus of thalamus for treatment of refractory epilepsy. Epilepsia 51, 899-908, 


doi: 10.1111/j.1528-1167.2010.02536.x (2010). 


. De Herdt, V. et al. Vagus nerve stimulation for refractory epilepsy: a Belgian multicenter study. Eur. J. Paediatr. Neurol. 11, 261-269, 


doi: 10.1016/j.ejpn.2007.01.008 (2007). 


. DeGiorgio, C. M. et al. Randomized controlled trial of trigeminal nerve stimulation for drug-resistant epilepsy. Neurology 80, 


786-791, doi: 10.1212/WNL.0b013e318285cl 1a (2013). 


. Fanselow, E. E., Reid, A. P. & Nicolelis, M. A. Reduction of pentylenetetrazole-induced seizure activity in awake rats by seizure- 


triggered trigeminal nerve stimulation. J. Neurosci. 20, 8160-8168 (2000). 


. Bergey, G. K. et al. Long-term treatment with responsive brain stimulation in adults with refractory partial seizures. Neurology 84, 


810-817, doi: 10.1212/WNL.0000000000001280 (2015). 


. Morgante, L. et al. How many parkinsonian patients are suitable candidates for deep brain stimulation of subthalamic nucleus? 


Results of a questionnaire. Parkinsonism Relat. Disord. 13, 528-531, doi: 10.1016/j.parkreldis.2006.12.013 (2007). 


. Morrell, M. J. & Group, R. N. S. S. i. E. S. Responsive cortical stimulation for the treatment of medically intractable partial epilepsy. 


Neurology 77, 1295-1304, doi: 10.1212/WNL.0b013e3182302056 (2011). 


. Vagus Nerve Stimulation Study Group. A randomized controlled trial of chronic vagus nerve stimulation for treatment of medically 


intractable seizures. Neurology 45, 224-230 (1995). 


. Shealy, C. N., Mortimer, J. T. & Reswick, J. B. Electrical inhibition of pain by stimulation of the dorsal columns: preliminary clinical 


report. Anesth. Analg. 46, 489-491 (1967). 


. Fuentes, R., Petersson, P., Siesser, W. B., Caron, M. G. & Nicolelis, M. A. Spinal cord stimulation restores locomotion in animal 


models of Parkinson’s disease. Science 323, 1578-1582, doi: 10.1126/science.1164901 (2009). 


. Yadav, A. P. et al. Chronic spinal cord electrical stimulation protects against 6-hydroxydopamine lesions. Sci. Rep. 4, 3839, doi: 


10.1038/srep03839 (2014). 


. Santana, M. B. et al. Spinal cord stimulation alleviates motor deficits in a primate model of Parkinson disease. Neuron 84, 716-722, 


doi: 10.1016/j.neuron.2014.08.061 (2014). 


. Agari, T. & Date, I. Spinal cord stimulation for the treatment of abnormal posture and gait disorder in patients with Parkinson's 


disease. Neurol. Med. Chir. (Tokyo). 52, 470-474 (2012). 


. Fenelon, G. et al. Spinal cord stimulation for chronic pain improved motor function in a patient with Parkinson's disease. 


Parkinsonism Relat. Disord. 18, 213-214, doi: 10.1016/j.parkreldis.2011.07.015 (2012). 


. Zhang, T. et al. Pre-seizure state identified by diffuse optical tomography. Sci. Rep. 4, 3798, doi: 10.1038/srep03798 (2014). 
. Jiao, J., Jensen, W., Harreby, K. R. & Sevcencu, C. The Effect of Spinal Cord Stimulation on Epileptic Seizures. Neuromodulation 19, 


154-160, doi: 10.1111/ner.12362 (2016). 


. Loscher, W. Critical review of current animal models of seizures and epilepsy used in the discovery and development of new 


antiepileptic drugs. Seizure 20, 359-368, doi: 10.1016/j.seizure.2011.01.003 (2011). 


. Berenyi, A., Belluscio, M., Mao, D. & Buzsaki, G. Closed-loop control of epilepsy by transcranial electrical stimulation. Science 337, 


735-737, doi: 10.1126/science. 1223154 (2012). 


. Schevon, C. A. et al. Evidence of an inhibitory restraint of seizure activity in humans. Nat.Commun. 3, 1060, doi: 10.1038/ 


ncomms2056 (2012). 


. Christensen, J., Kjeldsen, M. J., Andersen, H., Friis, M. L. & Sidenius, P. Gender differences in epilepsy. Epilepsia 46, 956-960, doi: 


10.1111/j.1528-1167.2005.51204.x (2005). 


. Harreby, K. R., Sevcencu, C. & Struijk, J. J. The effect of spinal cord stimulation on seizure susceptibility in rats. Neuromodulation 14, 


111-116; discussion 116, doi: 10.1111/j.1525-1403.2010.00320.x (2011). 


. Erkec, O. E. & Arihan, O. Pentylenetetrazole Kindling Epilepsy Model. Epilepsi 21, 6-12, doi: 10.5505/epilepsi.2015.08108 (2015). 
. Corda, M. G. et al. Pentylenetetrazol-induced kindling in rats: effect of GABA function inhibitors. Pharmacol. Biochem. Behav. 40, 


329-333 (1991). 


. Ilhan, A., Iraz, M., Kamisli, S. & Yigitoglu, R. Pentylenetetrazol-induced kindling seizure attenuated by Ginkgo biloba extract (EGb 


761) in mice. Prog Neuropsychopharmacol Biol. Psychiatry 30, 1504-1510, doi: 10.1016/j.pnpbp.2006.05.013 (2006). 


. Davoudi, M., Shojaei, A., Palizvan, M. R., Javan, M. & Mirnajafi-Zadeh, J. Comparison between standard protocol and a novel 


window protocol for induction of pentylenetetrazol kindled seizures in the rat. Epilepsy Res 106, 54-63, doi: 10.1016/j. 
eplepsyres.2013.03.016 (2013). 


. Fanselow, E. E., Sameshima, K., Baccala, L. A. & Nicolelis, M. A. Thalamic bursting in rats during different awake behavioral states. 


Proc. Natl. Acad. Sci. USA 98, 15330-15335, doi: 10.1073/pnas.261273898 (2001). 


. Nicolelis, M. A., Baccala, L. A., Lin, R. C. & Chapin, J. K. Sensorimotor encoding by synchronous neural ensemble activity at 


multiple levels of the somatosensory system. Science 268, 1353-1358 (1995). 


. Gervasoni, D. et al. Global forebrain dynamics predict rat behavioral states and their transitions. J. Neurosci. 24, 11137-11147, doi: 


10.1523/JNEUROSCI.3524-04.2004 (2004). 


. Paz, J. T. et al. Closed-loop optogenetic control of thalamus as a tool for interrupting seizures after cortical injury. Nat. Neurosci. 16, 


64-70, doi: 10.1038/nn.3269 (2013). 


. Schlaier, J. R. et al. Effects of spinal cord stimulation on cortical excitability in patients with chronic neuropathic pain: a pilot study. 


Eur. J. Pain 11, 863-868, doi: 10.1016/j.ejpain.2007.01.004 (2007). 


. Fanselow, E. E. & Nicolelis, M. A. Behavioral modulation of tactile responses in the rat somatosensory system. J. Neurosci. 19, 


7603-7616 (1999). 


SCIENTIFIC REPORTS | 6:32814 | DOI: 10.1038/srep32814 8 


www.nature.com/scientificreports/ 


35. Salanova, V. et al. Long-term efficacy and safety of thalamic stimulation for drug-resistant partial epilepsy. Neurology 84, 1017-1025, 
doi: 10.1212/WNL.0000000000001334 (2015). 

36. Englot, D. J., Rolston, J. D., Wright, C. W., Hassnain, K. H. & Chang, E. F. Rates and Predictors of Seizure Freedom With Vagus Nerve 
Stimulation for Intractable Epilepsy. Neurosurgery, doi: 10.1227/NEU.0000000000001165 (2015). 

37. Zare, M. et al. Trigeminal nerve stimulation: A new way of treatment of refractory seizures. Adv. Biomed. Res. 3, 81, doi: 
10.4103/2277-9175.127994 (2014). 

38. Kahlow, H. & Olivecrona, M. Complications of vagal nerve stimulation for drug-resistant epilepsy: a single center longitudinal study 
of 143 patients. Seizure 22, 827-833, doi: 10.1016/j.seizure.2013.06.011 (2013). 

39. Hayek, S. M., Veizi, E. & Hanes, M. Treatment-Limiting Complications of Percutaneous Spinal Cord Stimulator Implants: A Review 
of Eight Years of Experience From an Academic Center Database. Neuromodulation 18, 603-608; discussion 608-609, doi: 10.1111/ 
ner.12312 (2015). 

40. Dzirasa, K. et al. Impaired limbic gamma oscillatory synchrony during anxiety-related behavior in a genetic mouse model of bipolar 
mania. J. Neurosci. 31, 6449-6456, doi: 10.1523/JNEUROSCI.6144-10.2011 (2011). 

41. Dzirasa, K. et al. Dopaminergic control of sleep-wake states. J. Neurosci. 26, 10577-10589, doi: 10.1523/JNEUROSCI.1767-06.2006 
(2006). 

42. Cicurel, R. & Nicolelis, M. A. L. The relativistic brain: how it works and why it cannot by simulated by a Turing machine 1.1 edn, ISBN: 
1511617020 (Kios Press, 2015). 

43. Nicolelis, M. A. L. Beyond boundaries: the new neuroscience of connecting brains with machines-and how it will change our lives 1st 
edn, (Times Books/Henry Holt and Co., 2011). 

44. Paxinos, G. & Watson, C. The rat brain in stereotaxic coordinates. 4th edn, (Academic Press, 1998). 

45. Nicolelis, M. A. L. Methods for neural ensemble recordings. 2nd edn, (CRC Press, 2008). 

46. Pais-Vieira, M., Chiuffa, G., Lebedev, M., Yadav, A. & Nicolelis, M. A. Building an organic computing device with multiple 
interconnected brains. Sci. Rep. 5, 11869, doi: 10.1038/srep11869 (2015). 

47. Pais-Vieira, M., Lebedev, M., Kunicki, C., Wang, J. & Nicolelis, M. A. A brain-to-brain interface for real-time sharing of sensorimotor 
information. Sci. Rep. 3, 1319, doi: 10.1038/srep01319 (2013). 


Acknowledgements 

We are grateful for the assistance from Jim Meloy for the design and production of the multielectrode arrays as 
well as setup development and maintenance, Laura Oliveira, Terry Jones, and Susan Halkiotis for administrative 
assistance and preparation of the manuscript. This work was funded by a grant from The Hartwell Foundation. 


Author Contributions 

M.P.-V., A.P.Y., M.L. and M.A.L.N. designed the experiments; M.P-V., A.P.Y., D.M., A.S. and D.G. performed 
the experiments; M.P-V., A.P.Y. and M.A.L.N. analyzed the data; M.P-V., A.P.Y., M.L. and M.A.L.N. wrote the 
manuscript. 


Additional Information 
Supplementary information accompanies this paper at http://www.nature.com/srep 


Competing financial interests: The authors declare no competing financial interests. 


How to cite this article: Pais- Vieira, M. et al. A Closed Loop Brain-Machine Interface For Epilepsy Control 
Using Dorsal Column Electrical Stimulation. Sci. Rep. 6, 32814; doi: 10.1038/srep32814 (2016). 


This work is licensed under a Creative Commons Attribution 4.0 International License. The images 

Sal or other third party material in this article are included in the article’s Creative Commons license, 
unless indicated otherwise in the credit line; if the material is not included under the Creative Commons license, 
users will need to obtain permission from the license holder to reproduce the material. To view a copy of this 
license, visit http://creativecommons.org/licenses/by/4.0/ 


© The Author(s) 2016 


SCIENTIFIC REPORTS | 6:32814 | DOI: 10.1038/srep32814 9 


Creating a New Sense: The Infrared Rat Studies 


157 


IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 20, NO. 1, JANUARY 2012 85 


Virtual Active Touch Using Randomly Patterned 
Intracortical Microstimulation 


Joseph E. O’Doherty, Mikhail A. Lebedev, Zheng Li, and Miguel A. L. Nicolelis 


Abstract—Intracortical microstimulation (ICMS) has promise 
as a means for delivering somatosensory feedback in neuropros- 
thetic systems. Various tactile sensations could be encoded by 
temporal, spatial, or spatiotemporal patterns of ICMS. However, 
the applicability of temporal patterns of ICMS to artificial tactile 
sensation during active exploration is unknown, as is the minimum 
discriminable difference between temporally modulated ICMS 
patterns. We trained rhesus monkeys in an active exploration task 
in which they discriminated periodic pulse-trains of ICMS (200 Hz 
bursts at a 10 Hz secondary frequency) from pulse trains with the 
same average pulse rate, but distorted periodicity (200 Hz bursts 
at a variable instantaneous secondary frequency). The statistics of 
the aperiodic pulse trains were drawn from a gamma distribution 
with mean inter-burst intervals equal to those of the periodic 
pulse trains. The monkeys distinguished periodic pulse trains 
from aperiodic pulse trains with coefficients of variation 0.25 or 
greater. Reconstruction of movement kinematics, extracted from 
the activity of neuronal populations recorded in the sensorimotor 
cortex concurrent with the delivery of ICMS feedback, improved 
when the recording intervals affected by ICMS artifacts were 
removed from analysis. These results add to the growing evidence 
that temporally patterned ICMS can be used to simulate a tactile 
sense for neuroprosthetic devices. 


Index Terms—Bidirectional interface, brain—machine interface, 
intracortical microstimulation, neural prosthesis. 


I. INTRODUCTION 


tems for the restoration of hearing [1], [2] and vision 
(3]-[8] have been investigated for several decades. Interest 
in neuroprosthetic devices that combine both motor and sen- 
sory components has developed more recently [9]-[14]. One 
example of a bidirectional neuroprosthesis is a robotic limb 


S ENSORY neuroprostheses and sensory substitution sys- 


Manuscript received February 01, 2011; revised April 30, 2011, June 
16, 2011; accepted July 03, 2011. Date of publication December 27, 2011; 
date of current version January 25, 2012. This work was supported in part 
by the Defense Advanced Research Projects Agency (DARPA) under Grant 
N66001-06-C-2019, in part by the Telemedicine and Advanced Technology Re- 
search Center (TATRC) under Grant W81XWH-08-2-0119, and in part by the 
National Institutes of Health through NICHD/OD under Grant RC1HD063390. 
The work of M. A. L. Nicolelis was supported by the NIH Director’s Pioneer 
Award Program under Grant DP10D006798. 

J. E. O’Doherty is with the Department of Physiology and the W. M. Keck 
Foundation Center for Integrative Neuroscience, University of California, San 
Francisco, CA 94143 USA (e-mail: joeyo@phy.ucsf.edu). 

M. A Lebedev and Z. Li are with the Department of Neurobiology and 
the Center for Neuroengineering, Duke University, Durham, NC 27710 USA 
(e-mail: lebedev @neuro.duke.edu; zheng @cs.duke.edu). 

M. A. L. Nicolelis is with the Departments of Neurobiology, Biomedical En- 
gineering, Psychology, and the Center for Neuroengineering, Duke University, 
Durham, NC 27710 USA (e-mail: nicoleli@neuro.duke.edu). 

Color versions of one or more of the figures in this paper are available online 
at http://ieeexplore.ieee.org. 

Digital Object Identifier 10.1109/TNSRE.2011.2166807 


controlled by brain activity while sensory information from 
prosthetic sensors is delivered to somatosensory areas of the 
brain [15], [16]. Other possible implementations include sen- 
sorized neuroprostheses for the restoration of bipedal walking 
[17] and putative systems combining both speech production 
[18], [19] and hearing [1], [2], [20]. 

In recent years, we have been studying intracortical micros- 
timulation (ICMS) delivered through microelectrode arrays 
chronically implanted in the primary somatosensory cortex 
(Sl) as a means of adding a somatosensory feedback loop 
to a brain—machine interface (BMI) [9], [10], [21]. Taken 
together with previous work showing that primates [22]—[24] 
and rodents [25]—-[28] can discriminate ICMS patterns, there is 
growing evidence that ICMS of S1 could equip neuroprosthetic 
limbs with the sense of touch. 

One of the neuroprosthetic devices that we envision in the fu- 
ture is a BMI-operated robotic arm that is equipped with touch 
sensors [9], [15], [16]. In such a sensorized neuroprosthesis, the 
touch sensors would detect instances when the arm interacts 
with external objects sending signals to the brain in the form 
of ICMS. We have suggested that long-term operation of such a 
system, which we call a brain—machine-brain interface (BMBD), 
could result in the incorporation of the prosthesis into the brain’s 
representation of the body, so that the artificial limb starts to act 
and feel as belonging to the subject [15]. Notwithstanding ini- 
tial encouraging results [9], [10], it is unclear whether ICMS 
would be sufficient to reproduce the rich sensory information of 
the world of touch [29]. 

In particular, it is not well understood which kinds of ICMS 
patterns are most useful for virtual active touch. Previously, 
we have shown that both New World [21] and Old World 
monkeys [10] can discriminate temporal ICMS patterns ap- 
plied to S1 that consist of short (50-300 ms) high-frequency 
(100-400 Hz) pulse-trains presented at a lower secondary 
frequency (2-10 Hz). In these experiments, ICMS served as 
a cue that instructed the direction of reach. These patterns of 
ICMS could, in principle, mimic a wide variety of tactile inputs, 
especially when combined with spatial encoding [21]. Mod- 
ulations of sensory inputs in this frequency range correspond 
to the sensation of flutter [30]—[32]. These timescales are also 
similar to neuronal modulations involved in texture encoding in 
the somatosensory system [33]-[36], which makes such ICMS 
patterns worthy candidates for exploration. 

In this study, we examined the ability of rhesus monkeys to 
discriminate a range of temporal ICMS patterns applied to S1 in 
the context of an active exploration task in which ICMS mim- 
icked the tactile properties of virtual objects. We manipulated 
the ICMS patterns in a graded fashion, modulating the degree 
of periodicity of the pulse-trains while maintaining a constant 
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average pulse rate. We sought to determine the minimal pertur- 
bation of the periodic pattern that the monkeys could discrimi- 
nate. The degree of randomness (as quantified by the coefficient 
of variation, CV) was varied from trial to trial, which allowed us 
to quantify the monkeys’ sensitivity to ICMS frequency mod- 
ulations. Concurrently with ICMS delivery, we recorded from 
large populations of cortical neurons using multielectrode im- 
plants. Kinematics of reach movements were extracted from this 
large-scale activity offline to estimate the accuracy of a BMBI 
with a somatosensory feedback loop that transmits aperiodic 
ICMS patterns. 

This investigation of sensitivity to ICMS periodicity in S1 
was motivated by a possible application in neuroprosthetic 
limbs. We expect that the patterns of ICMS triggered by the 
interaction of an upper-limb neuroprosthesis with objects in the 
environment could be highly irregular. The precise temporal 
structure of such patterns would depend on the interaction 
of touch sensors in the robotic prosthesis with the specific 
surface structure of the manipulated objects and on the specific 
exploratory movements used by the individual to interact with 
the objects. Therefore, by knowing the limits of the nervous 
system in discriminating aperiodic ICMS patterns, we can 
infer a principled upper bound on the maximum fidelity touch 
sensor that could be used in a neuroprosthesis, beyond which 
no additional function would be restored. 

This study builds on the results obtained by Romo et al. about 
ICMS of S1 [22], [23], [31] and our own previous work [9], [21]. 
One notable difference between the temporal ICMS patterns im- 
plemented here and those used by Romo et al. is that the aperi- 
odic patterns of ICMS that we used had the same mean pulse in- 
terpulse intervals as the periodic comparison ICMS pulse trains. 
Thus the average number of pulses in a pulse train was the 
same for both periodic and aperiodic patterns. This allowed us to 
probe S1 sensitivity to the temporal structure of ICMS without 
the confound of average stimulus intensity. Romo ef al. used 
periodic pulse trains with different frequencies [22], which left 
open the possibility that some of their results could be explained 
by differences in average ICMS intensity. 

Another major difference between this study and previous 
studies of ICMS-evoked S1 sensations in primates is that the 
ICMS patterns employed here were used in an active-explo- 
ration paradigm in which ICMS was used to simulate the tactile 
properties of virtual objects. Our monkeys explored the virtual 
objects, making self-paced exploratory movements, and decided 
which objects to explore, in what order, and for how long. This 
is amore realistic model of a clinical somatosensory neuropros- 
thesis than previous designs. 


II. METHODS 


A. Implants 


The experiments were conducted in two rhesus monkeys (M 
and N) chronically implanted with multielectrode arrays in sev- 
eral cortical areas following our implantation methods [37]. We 
used this same electrode array design for both large-scale neural 
recordings and ICMS delivery [9], [10]. Each monkey received 
four 96-channel microelectrode arrays placed in the arm and leg 


Feedback zone 
Target zone 
wate Avatar 
—ttt}— 
ICMS 
(c) 
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1.Go 2. Explore 3. Select 
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Fig. 1. Implants and task paradigm. (a) The monkeys were implanted with 
microwire arrays targeting M1 and S1 of the upper and lower limbs. (b) 
Channels used for stimulation with monkey M are accented in red. (c) Objects 
on the screen consisted of a central response zone surrounded by a peripheral 
feedback zone. Movement of the avatar though the feedback or response zone 
triggered the delivery of ICMS pulse trains. (d). Monkeys initiated a trial 
by holding the avatar in the center of the screen until two peripheral objects 
appeared (500-1000 ms, random per trial; left sub-panel). Next, the monkeys 
freely explored the objects (middle sub-panel). Finally, an object was selected 
by holding the avatar within the response zone for 2000 ms (right sub-panel). 


representation areas in sensorimotor cortex [Fig. 1(A)]. Each 
hemisphere was implanted with two arrays: one in the arm rep- 
resentation and one in the leg representation. Within each array, 
electrodes were grouped in two 4x4 uniformly spaced grids of 
electrode triplets. The electrodes within each triplet had dif- 
ferent lengths, staggered at 300 jm intervals. One grid was 
aligned over primary motor cortex (M1) and the other over S1. 
The monkeys were implanted for a series of studies beyond 
those described here. For the purpose of this study, we recorded 
neuronal activity from the right hemisphere arm arrays while the 
monkeys performed a manual task with their left hands. Stim- 
ulation was applied to the right hemisphere arm subdivision of 
S1 in monkey M and the right hemisphere leg subdivision of S1 
in monkey N. All animal procedures were performed in accor- 
dance with the National Research Council’s Guide for the Care 
and Use of Laboratory Animals and were approved by the Duke 
University Institutional Animal Care and Use Committee. 


B. Behavioral Task 


The monkeys were trained in a reaching task in which they 
manipulated a hand-held joystick to move a virtual reality arm 
(avatar) displayed on a computer screen [Fig. 1(C) and (D)]. 
The monkeys reached with the avatar arm towards screen ob- 
jects and searched for an object with a particular artificial tex- 
ture indicated by ICMS of S1. The objects were circular in shape 
and visually identical. Each monkey was previously trained in 
other variants of this task. In this study, the monkeys were shown 
two objects, one of which was associated with a periodic ICMS 
pattern and the other with an aperiodic pattern [Fig. 2(A)]. The 
monkeys were rewarded for selecting the object paired with 
periodic ICMS. The objects appeared at different locations on 
the screen, with the constraint that the distance from the screen 
center to each object was fixed, and the angle between the ob- 
jects was 180° (i.e., centrally symmetric). If the correct object 
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Fig. 2. Example aperiodic ICMS pulse trains. (a) Raster indicates the range of 
variability of inter-burst intervals (CV = 0.8, cyan; CV = 0.5, blue; CV = 
0.25, green; CV = 0.05, red; CV = 0, black). Note that each vertical line 
indicates a short burst of ICMS, not a single pulse. (a) Distributions of inter-burst 
intervals corresponding with the rasters shown in (a). 


was selected, the monkeys were rewarded with a drop of fruit 
juice. 

Each trial commenced when the monkey grasped the joystick 
with its left hand. At this point, a circular target appeared in the 
center of the screen. The monkey placed the avatar arm on that 
center target for 0.5—-1.0 s. Then, the central target disappeared 
and two peripheral objects appeared. Each consisted of a central 
response zone and a peripheral feedback zone [Fig. 1(C)]. When 
the avatar hand entered the feedback or response zones, ICMS 
pulse trains were delivered to S1. A trial was concluded with 
a reward when the monkey placed the avatar hand within the 
response zone of the correct object for 2 s; no reward was deliv- 
ered if the incorrect object was selected. The monkeys were per- 
mitted to explore the virtual objects in any sequence, but the trial 
ended when they stayed over an object’s response zone longer 
than the hold period of 2 s. Then a 0.5 s delay was issued before 
the next trial began. 


C. ICMS Patterns 


ICMS trains consisted of symmetric [38], biphasic, charge- 
balanced pulses of ICMS delivered in a bipolar fashion through 
adjacent pairs of microwires [9], [21]. For monkey M, the an- 
odic and cathodic phases of stimulation each had amplitudes of 
150 A and pulse widths of 105 jus; for monkey N, 150 A and 
200 us, respectively. The anodic and cathodic phases were sep- 
arated by 25 jus. 

ICMS was delivered to different subdivisions of S1 for each 
monkey. For monkey M, the hand representation area of SI was 
used as the target for ICMS, so that the monkey experienced 
putative sensations in its hand [Fig. 1(B)]. For monkey N, ICMS 
was applied to the thigh representation area of S1. Two electrode 
pairs were used for each animal. 


The temporal pattern of ICMS consisted of 200 Hz pulse 
trains delivered for 50 ms and presented at a lower secondary 
frequency. The secondary frequency for the rewarded artificial 
texture was a constant 10 Hz. For the unrewarded textures, the 
timing of the ICMS bursts was aperiodic. The interval between 
each aperiodic burst was a random variable drawn from a 
gamma distribution of instantaneous inter-burst intervals [39] 


ok le-6 


Tim 6F 


where f is the probability density function, x is the inter-burst 
interval, k; is the shape parameter, 6 is the scale parameter, and 
T is the Gamma function. We computed the shape and scale pa- 
rameters as a function of the mean inter-pulse interval, j:, and 
the coefficient of variation, CV, the ratio of the standard devia- 
tion to the mean 


f(a5k; 6) = () 


Cv? 
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This allowed the construction of aperiodic pulse trains with 
inter-burst intervals equal in expectation to the periodic pulse 
trains while giving control over the degree of aperiodicity: the 
higher the CV, the more aperiodic the pulse-train. A pulse train 
with a CV of zero was equivalent to the periodic, rewarded 
pattern. The average number of ICMS pulses per unit time was 
the same for the periodic and aperiodic patterns. Examples of 
pulse trains with different CVs are shown in Fig. 2. 


D. Artifact Suppression 


An important question arising from the use of ICMS for sen- 
sory feedback is whether the stimulation causes artifacts in cor- 
tical neural ensemble recordings and how these artifacts can 
be dealt with to minimize their impact on BMI operations. To 
address this question, we processed the neural recordings by 
removing (blanking) a window of neural activity immediately 
subsequent to each pulse of ICMS and then performed decoding 
with the processed data. By systematically varying the length of 
the blanking intervals we could determine the amount of artifact 
removal that produced the most accurate movement reconstruc- 
tions. 

Artifact removal was implemented as follows. The stim- 
ulation artifacts had stereotypical shapes when recorded by 
the spike acquisition system, and we could reliably detect 
artifacts by using spike-sorting templates that matched the 
artifact shapes, allowing us to determine the precise time of 
each stimulation pulse. We then ignored all spiking on every 
channel for x milliseconds after each stimulation pulse, where 
x varied in value from 0 to 10, in integer steps. This was done 
by first counting spikes in | ms nonoverlapping time windows 
(i.e., binning at 1 ms resolution). We then zeroed the spike 
counts in bins that were equal to or less than x ms after the 
stimulation pulse. For example, for 7 = 3, we zeroed the bins 
at t,t + 1, t + 2, and ¢ + 3, where ¢ is the | ms bin of the 
stimulation pulse [see Fig. 6(A)]. For 7 = 0, we zeroed the 
bin at the stimulation pulse only. Then, we summed adjacent 
bins to produce spike counts in 100 ms nonoverlapping bins for 
our decoders. For this last step, we adjusted the spike counts 
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Fig. 3. Summary of the behavioral performance of monkeys M (circles) and 
N (diamonds) for 10 sessions as they learned the task. Each symbol shows the 
mean performance for the session. Filled symbols depict sessions with perfor- 
mance significantly different from chance (chance level of 50%; P < 0.05, 
one-sided binomial test). Curves are the sigmoidal lines of best fit. 


by multiplying by the quantity 100/(100 — nplankea), where 
Mblanked 1S the number of zeroed 1 ms bins in the 100 ms bin. 
This operation preserves, in expectation, the number of spikes 
in each 100 ms bin, by performing extrapolation. 


E. Kinematics Extraction 


The X and Y position of the avatar was extracted from cor- 
tical activity using a fifth-order unscented Kalman filter [40] and 
Wiener filter [37]. For both algorithms, we evaluated the de- 
coding accuracy after artifact blanking. We performed two-fold 
cross-validation with each algorithm on 26 sessions, 13 from 
each monkey. For the unscented Kalman filter, we used a tuning 
model with linear weights for position, velocity, distance from 
center of workspace, and magnitude of velocity. The unscented 
Kalman filter had three future taps, two past taps, and one tap 
in the movement model (see [40] for details). The tuning model 
weights were fit with adaptive ridge regression [41], with the 
ridge parameter found by cross-validation on the training data. 
For the Wiener filter, we used 10 taps of spiking history and pre- 
dicted the position only. The Wiener coefficients were fit using 
ridge regression with the ridge parameter found by cross-vali- 
dation on the training data. 


Ill. RESULTS 


A. Learning 


Initially, both monkeys were required to discriminate be- 
tween the periodic (rewarded) ICMS pattern and an aperiodic 
pattern with a CV of 0.8. Each monkey learned this discrim- 
ination task in approximately eight daily sessions (Fig. 3). 
Monkey N stabilized at a performance level of approximately 
90% correctly executed trials, monkey M at an 85% level. 
These learning curves are consistent with our previous results 
on rhesus monkey learning with ICMS-instructed tasks [9]. 
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Fig. 4. Psychometric curves for different coefficients of variation on the aperi- 
odic pulse trains. (a) Mean performance at differentiating periodic versus aperi- 
odic ICMS pulse trains as a function of CV for monkey N. Each symbol repre- 
sents the mean performance across sessions; error bars indicate 95% confidence 
intervals. Curves are the sigmoidal lines of best fit. Symbols are as in Fig. 3. (b) 
Same as (a), but for monkey M. 


B. Psychometrics 


After both monkeys learned to discriminate periodic ICMS 
from aperiodic with a CV of 0.8, we began to vary the CV of 
the aperiodic ICMS pattern on every trial. In these sessions, the 
distribution of CVs was picked so that for half of the trials the 
CV of the unrewarded object was greater or equal to 0.6. These 
sessions continued for two weeks, yielding a database for psy- 
chometric analysis. 

Psychometric curves (i.e., graphs showing the proportion of 
correctly performed trials as the function of CV, Fig. 4) indi- 
cated a clear dependency of discrimination accuracy on the de- 
gree of randomness of the comparison ICMS pattern. Perfor- 
mance stabilized for CVs higher than 0.8 and gradually de- 
creased for CVs lower than that value. The threshold CV for 
discrimination for both monkeys was 0.25. Below this value, 
the monkeys performed at chance levels. 


C. Active Exploration 


Discrimination of ICMS patterns was performed through ac- 
tive exploration: a monkey would probe the feedback zone of 
an object with the avatar to acquire an ICMS pattern and then 
either select that object if it perceived the ICMS pattern as peri- 
odic or explore the other object if the pattern was judged as dif- 
ferent from periodic. This active exploration was evident from 
an analysis of object exploration intervals [Fig. 5(A) and (B)]. 
We designated intervals during which the avatar hand continu- 
ously stayed over a given object as “visits.” For very low CVs, 
the statistics of visit durations were the same for periodic and 
aperiodic patterns [Fig. 5(A)]. 

The distribution of these intervals indicated short (less than 
2 s) exploratory visits and a prominent peak at 2 s that corre- 
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Fig. 5. Active exploration of the virtual objects as a function of CV. (a) His- 
tograms of visit durations to the rewarded object (CV = 0, black trace) versus 
the unrewarded object (CV = 0.05, red trace) for trials with that CV combina- 
tion. Visits are quantified as intervals when the avatar was continuously over the 
feedback zone or the response zone. Vertical line indicates the hold interval (2 s). 
(b) Same as (a), but for trials with unrewarded objects with aperiodic ICMS CVs 
of 0.8 (cyan trace). (c) Fraction of the short visits (less than 2 s) for the rewarded 
object (square symbols) versus the unrewarded object (triangular symbols) ex- 
pressed as a function of CV. Gray shaded zones correspond to the data in (a) 
and (b). 


sponded to selecting an object. This is because the monkey was 
required to hold the avatar hand over the response zone of the 
object for 2 s to obtain a reward (or to get a trial cancellation if 
the object was selected incorrectly). Accordingly, the peak at 2 s 
corresponded to visits for which the monkey selected a given ob- 
ject. Intervals longer than 2s were possible because visits com- 
prised the portion spent over the feedback zone (but outside of 
the response zone) as well as the time spent over the response 
zone. We called visits with durations less than 2 s short visits, 
and those with durations of 2 s or longer long visits. Both the 
short-visit portion of the distribution and the long-visit part were 
preserved for periodic ICMS (CV = 0, Fig. 5(A), black line) 
versus weakly aperiodic ICMS e.g., CV of 0.05 (Fig. 5(A), red 
line). 

The distributions of visit-durations were markedly different 
for higher CVs [Fig. 5(B)]. The distribution of visit-durations 
for aperiodic ICMS with a CV of 0.8 (cyan line) revealed a pre- 
dominance of short visits with an average duration of 0.8 s and 
a small proportion of long visits. For the periodic pattern (black 
line), the distribution showed the predominance of long visits. 
These data indicate that it took the monkey on average 0.8 s to 
recognize the unrewarded aperiodic ICMS pattern and to switch 
to the correct object (periodic pattern) when sufficiently aperi- 
odic ICMS patterns were used. 

The change in monkey exploratory behavior for different de- 
grees of ICMS-pattern aperiodicity is clear from the statistics of 
visits, expressed as the proportion of short visits normalized by 
the total number of visits [Fig. 5(C)]. When a monkey touched 
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Fig. 6. Exploration of blanking intervals. (a) Schematic of the blanking proce- 
dure. Spikes and ICMS pulses were categorized into | ms bins. For each bin con- 
taining an ICMS pulse, that bin and a variable number of bins (three shown here) 
were blanked subsequently. (b). Mean movement reconstruction accuracy as a 
function of blanking intervals for both monkeys (symbols as in Fig. 3) and both 
algorithms (Wiener filter, WF, dashed lines; unscented Kalman filter, UKF, solid 
lines). Bars indicate standard error. Shaded region corresponds to no blanking. 


an object associated with an aperiodic pattern (Fig. 5(C), trian- 
gles), it tended to make more short visits than when the monkey 
touched an object associated with a periodic pattern (squares). 
For high CVs (CV greater than 0.7), the proportion of short 
visits constituted approximately 80% of the total number of 
visits. For lower CVs, this value decreased, indicating that the 
monkey made a decision to stay on the unrewarded object more 
often. 


D. Kinematics Extraction 


Fig. 6 shows the average accuracy for the extraction of avatar 
position from cortical ensemble activity for different lengths of 
artifact blanking intervals. Consistent with our previous results 
[40], the unscented Kalman filter consistently outperformed the 
Wiener filter. For monkey M, the peak accuracy was 5.4 + 
0.17 dB (mean + standard error) for the unscented Kalman filter 
and 4.0 + 0.14 dB for the Wiener filter. For monkey N, these 
values were 2.9+0.16 dB and 2.5+0.14 dB, respectively. These 
accuracy values are within the range that we typically observe 
for BMI predictions [9], [17], [41]. 


90 IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 20, NO. 1, JANUARY 2012 


Both algorithms benefited somewhat from artifact blanking, 
more so for monkey M. For monkey M, maximum accuracy was 
achieved with 5 ms of artifact blanking for both the Weiner filter 
and the unscented Kalman filter. For monkey N, maximum ac- 
curacy was achieved with 2 ms of artifact blanking for both de- 
coders. These values reflect the difference in artifact duration 
and amplitude in two monkeys. The artifacts were more promi- 
nent and of longer duration in monkey M because of the close 
proximity of the stimulation site (hand representation of $1) to 
the area where neuronal activity was collected (arm represen- 
tation of M1 and S1). The artifacts were smaller and of shorter 
duration for monkey N, which received stimulation in the leg 
representation area of S1 with recordings performed in the arm 
representation area. 

Curiously, the performance of the unscented Kalman filter 
was slightly better for the no blanking condition than for 0 ms 
of blanking. This was because the recording channels that de- 
tected ICMS artifacts occasionally recorded additional mechan- 
ical artifacts related to monkey head movements. Apparently, 
the filter could utilize these mechanical artifacts that influenced 
the spike recording channels (blanked them or introduced erro- 
neous spikes) to improve predictions, and its performance was 
very slightly reduced when these artifacts were removed. This 
underscores the importance of registering the artifacts and re- 
moving them to minimize their influence on the filter perfor- 
mance. 

To quantify the decrease in predictions caused by the presence 
of ICMS artifacts, we recorded from both monkeys as they per- 
formed a center-out task without any ICMS. For this task, they 
had to move the avatar from the center of the screen to a single 
peripheral object and hold for 2 s. For monkey M, accuracy in 
this task was 23% higher than during the ICMS sessions with the 
unscented Kalman filter and 32% higher with the Wiener filter. 
For monkey N, these values were 2.7% and 27%, respectively. 
Thus, the artifacts worsened the predictions, even after optimal 
blanking, but still within a tolerable range. 


IV. DISCUSSION 


This study continued our work on the development of an arti- 
ficial somatosensory channel for BMIs [9], [10], [21]. Monkeys 
scanned virtual objects with an avatar hand and discriminated 
their artificial textures as represented by temporal patterns of 
ICMS. This paradigm models the requirements of a clinically 
relevant neuroprosthetic arm sensorized with an artificial tactile 
channel. Such a neuroprosthetic arm could be used to touch ex- 
ternal objects and estimate their tactile properties (roughness/ 
smoothness, hardness/softness, wetness/dryness, temperature) 
using sensors on the prosthetic hand. The transmission of this 
information to the nervous system is a difficult problem because 
of the artificial nature of the stimulation methods. We explored 
the capability of temporally patterned ICMS as a way to deliver 
somatosensory feedback to the brain by parametrically varying 
the degree of randomness of ICMS trains. Monkeys learned 
to distinguish regular ICMS patterns from irregular ones, a re- 
sult which suggests that they were able to discriminate the fine 
temporal structure of ICMS trains. Irregular bursts of sensory 
discharges are expected to occur in practical neuroprostheses, 
when the prosthesis interacts with realistically textured objects. 


A neuroprosthetic hand used to scan a ridged surface, for ex- 
ample, would generate an ICMS burst each time a ridge inter- 
acts with the prosthetic sensor. In this setting, the degree of pe- 
riodicity of ICMS pulse trains could inform the prosthesis user 
about the regularities or irregularities of an object’s material or 
shape. 

Our results complement previous work on ICMS frequency 
discrimination conducted by Romo et al. who trained their mon- 
keys to discriminate periodic ICMS pulse trains [22]-[24] and to 
discriminate the mean rate of aperiodic pulse trains [22], [42]. 
Our study expanded the range of temporal patterns that could 
be represented by ICMS of S1 by changing the regularity of 
the secondary frequency. Moreover, ICMS in our experiments 
served as somatosensory feedback during virtual active touch, 
rather than merely a cue in a forced choice task as in the ma- 
jority of previous studies. The animals actively explored vir- 
tual objects with an avatar hand, spending similar times over 
these objects as would be needed for normal interaction with 
the environment. Additionally, chronically implanted electrodes 
were used for ICMS delivery, which allowed us to monitor long- 
term learning to utilize ICMS as sensory feedback. In previous 
studies, stimulating electrodes were often inserted in the brain 
anew during each daily session. Long-term usage of ICMS in 
the present experiments (as well as in previous experiments 
with the same monkeys) did not result in deterioration of per- 
formance, which indicates that the charge-balanced ICMS used 
here did not damage the electrodes or brain tissue, or that any 
such damage was below a threshold where it would begin to im- 
pact task performance. 

Our results show that monkeys detect distortions in the 10 Hz 
ICMS secondary frequency after random variations of that fre- 
quency exceeded 25%, that is, instantaneous frequency fluctu- 
ated from 7.5 to 12.5 Hz. This estimate of the detection threshold 
can be used in future neuroprosthetic designs as a characteristic 
sensitivity value. Future studies should probe the sensitivity of 
discrimination to different primary and secondary frequencies. 
Additionally, spatiotemporal ICMS [21] and ICMS of different 
durations should be explored as ways to encode information in 
BMBI sensory channels. 

The interaction of a neuroprosthesis with realistically tex- 
tured objects in the natural world will inevitably result in a 
stream of temporally patterned sensory information. Either the 
user or the neuroprosthesis (or some combination thereof) will 
therefore need to deal with these signals. Texture analysis could 
be delegated, in part, to a shared control algorithm [43]. In this 
mode of operation, a sensation processor would analyze raw sig- 
nals from sensors on the prosthesis and interpret them in the 
context of how the neuroprosthetic device “skin” moved against 
the surface of textured objects. Simplified ICMS patterns—rep- 
resenting different classes of textures—could then be sent to 
the brain. Alternatively, ICMS could directly encode a signal 
representing both the spatiotemporal movements of the pros- 
thetic limb and the intrinsic microstructure of the material being 
touched. In this case, the temporal patterns of ICMS would have 
to be interpreted by the user in the context of the particular ex- 
ploration pattern used [44]. The choice of the encoding scheme 
will likely be dictated by the requirements of the specific neu- 
roprosthetic application. 
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It would be of interest for future studies to explore the optimal 
temporal properties of ICMS modulations at different $1 sites. 
Romo ef al. reported best results when they applied ICMS to 
rapidly adapting neurons in area 3b [22]. We stimulated in area 
1, where the distinction between rapidly adapting and slowly 
adapting categories of neurons is less clear. Additionally, we 
used multi-session training periods with chronically implanted 
electrodes, in contrast to Romo et al., who used independent 
stimulation sessions with acute electrodes. It is possible that 
that our longer training interval facilitated the discrimination ca- 
pacity of the monkeys. The distinction between different S1 lo- 
cations and the role of learning will need to be studied in more 
detail in future studies. 

BMBIs equipped with afferent ICMS feedback loops need 
to compensate for electrical artifacts produced by ICMS pulses 
that may interfere with neuronal recordings. In our previous 
BMBI designs, we either discounted the entire period of ICMS 
application [9] or used interleaved recording and ICMS delivery 
intervals [10]. These previous approaches limited the flexibility 
of ICMS delivery. In this study, we did not impose limitations 
on the timing of ICMS delivery and treated ICMS artifacts as 
they occurred. We found that blanking the periods after ICMS 
delivery by short intervals (2-5 ms) improved the accuracy of 
extraction of limb kinematics from neuronal activity. Overall 
accuracy of predictions was 20%-—30% less as compared to ses- 
sions in which ICMS was not used. Nonetheless, the predic- 
tions were still acceptable and within range of previously re- 
ported accuracy of BMI decoding. This result suggests that arti- 
fact blanking is practical for bidirectional neuroprostheses using 
irregular ICMS pulse trains. 

The precise character of perceptions evoked by periodic 
versus aperiodic patterns of ICMS will have be evaluated in 
human subjects [45]. There is a suggestion by Fridman ef al. 
that ICMS amplitude, pulse-width, and frequency all interact to 
contribute to a unitary perception of “perceived intensity” [46]. 
Therefore, one might argue that our monkeys discriminated 
the periodic and aperiodic pulse trains on the basis of their in- 
stantaneous peak intensities rather than their temporal patterns. 
However, this simple explanation is unlikely because the peak 
instantaneous frequency of ICMS was 200 Hz for both the 
periodic as well as the aperiodic artificial textures. Therefore, 
our results indicate that the monkeys must have been using a 
strategy beyond simply detecting the maximum instantaneous 
frequency. One possible neural implementation could employ 
a leaky integrator mechanism that detected variability of the 
ICMS secondary frequency by integrating neural responses to 
ICMS within an optimal time window, thus detecting transient 
increases in ICMS frequency. This and other alternative mech- 
anisms will need to be elucidated with future studies. 

Our current and previous [9], [10], [21] results suggest that 
new perceptions may evolve as subjects practice with ICMS. 
We observed that it took monkeys 1-2 weeks to start to under- 
stand ICMS, even if they were previously overtrained with a vi- 
brotactile variant in the same task. However, once they learned 
the first ICMS task, learning subsequent tasks took much less 
time. A virtual active touch setting where subjects evoke ICMS 
and associated sensations through their own actions [47]-[49] 
may contribute to shaping the artificial perception and lead to 


the development of anticipatory cortical modulations similar to 
corollary discharge [50]. 

The problem of artifacts will be compounded as multiple 
stimulation channels are employed with asynchronously deliv- 
ered pulses. Excessive masking of the recordings by ICMS ar- 
tifacts should be avoided in the design of such systems. In the 
future, the problem of artifacts [51], as well as the unreliable 
spatial extent of ICMS [52] could be mitigated by optogenetic 
stimulation [53]—[55]. 
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Artificial sensation via electrical or optical stimulation of brain sensory areas offers a promising treatment for sensory deficits. For 
a brain-machine- brain interface, such artificial sensation conveys feedback signals from a sensorized prosthetic limb. The ways 
neural tissue can be stimulated to evoke artificial sensation and the parameter space of such stimulation, however, remain largely 
unexplored. Here we investigated whether stochastic facilitation (SF) could enhance an artificial tactile sensation produced by 
intracortical microstimulation (ICMS). Two rhesus monkeys learned to use a virtual hand, which they moved with a joystick, to 
explore virtual objects on a computer screen. They sought an object associated with a particular artificial texture (AT) signaled by 
a periodic ICMS pattern delivered to the primary somatosensory cortex (S1) through a pair of implanted electrodes. During each 
behavioral trial, aperiodic ICMS (i.e., noise) of randomly chosen amplitude was delivered to $1 through another electrode pair 
implanted 1 mm away from the site of AT delivery. Whereas high-amplitude noise worsened AT detection, moderate noise clearly 
improved the detection of weak signals, significantly raising the proportion of correct trials. These findings suggest that SF could 


be used to enhance prosthetic sensation. 


Introduction 

Perhaps somewhat counterintuitively, combining informative 
signals with noise can enhance signal processing in biological 
systems (Moss et al., 2004). This phenomenon is referred to as 
stochastic facilitation (SF) or stochastic resonance (McDonnell 
and Ward, 2011). SF occurs when moderate levels of noise in- 
crease the number of neural threshold crossings, but also keep 
those crossings in synchrony with the signal (Moss et al., 2004). 
More intense noise masks the signal and worsens its detection. 
Examples of SF in human sensory processing include improved 
detection of visual stimuli (Kitajo et al., 2003), as well as mechan- 
ical stimuli applied to the fingers (Collins et al., 1996) and the foot 
sole (Wells et al., 2005). Similarly, SF has been reported for the 
cercal sensory system of crickets (Levin and Miller, 1996), CA1 
neurons in hippocampal slices from rats (Stacey and Durand, 
2001), and the spinal circuits of the cat (Martinez et al., 2007). 
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However, there is an ongoing debate as to whether SF serves any 
positive function in the neural circuitry (McDonnell and Abbott, 
2009; McDonnell and Ward, 2011). Additionally, it is unclear 
whether SF could be used in assistive devices for people with 
sensory disabilities, for example, systems for sensory substitution 
(Bach-y-Rita and Kercel, 2003) and neural prosthetics with arti- 
ficial sensation (Romo et al., 1998; O’Doherty et al., 2011). 

Recently we described a sensorized neural prosthetic, called 
BMBI (brain—machine—brain interface), where neuronal activity 
recorded in the primary motor cortex (M1) was used to control 
the movements of a virtual arm while artificial tactile feedback 
from such an actuator was generated by intracortical micro- 
stimulation (ICMS) patterns delivered directly to somatosensory 
cortex (S1) (O’Doherty et al., 2009, 2011, 2012). This paradigm 
opens the possibility of aiding future human users of neuropros- 
thetic limbs with sensory signals arising from a variety of limb 
sensors. However, to develop sensorized neural prostheses fur- 
ther, detailed exploration is needed into ICMS as the means to 
reproduce the exteroception and proprioception normally pro- 
vided by somatic sensation. Of particular interest is an inquiry 
into the processing of sensations evoked by ICMS when they are 
mixed with noisy inputs. For natural sensation, this problem is 
routinely solved by biological organisms in real-life environ- 
ments (Moss et al., 2004). For artificial sensation, however, the 
research is only starting (O’Doherty et al., 2012). In the present 
study, we hypothesized that SF could enhance detection of weak 
ICMS patterns under an active exploration paradigm (O’ Doherty 
et al., 2011; 2012). Accordingly, we delivered mixtures of stimu- 
lus and noise to S1 in different proportions and tested how their 
different combinations affected the subject’s active exploration of 
artificial textures (ATs). 
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Experimental setup and task schematics. A, Arrays of microelectrodes were implanted in $1 and M1. Stimulating electrodes for monkey M (inset) are highlighted in red (AT) and yellow 


(noise). B, Monkeys manipulated a joystick to move a virtual arm and reach toward objects on a computer screen. C, Experimental task sequence. In the illustrated case, the selection of the right-hand 
object resulted in reward. The activity of both AT and noise ICMS channels are shown below each box. 
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Figure 2. 


Detection of signal in noise and M1 modulations. Example traces of horizontal movement of the virtual hand for three trials with different noise levels: low (A), optimal (B), and high 


(Q). The colored horizontal bars represent the position of the targets (green, RAT; brown, NAT). In the trials shown in A and C, the monkey incorrectly selected the NAT, whereas in B it correctly picked 
the RAT and received a reward (indicated by a black arrow). The pulses delivered through both stimulating channels are shown. The RAT is subthreshold. The corresponding neuronal ensemble 
activity (monkey M, 120 units from M1) is shown in the bottom plots. Note that neuronal modulations reflect the speed and direction of arm movements. The color scale shows normalized firing rate 


(in Hz). 


Materials and Methods 


Two rhesus monkeys, M (male) and N (female), were chronically im- 
planted with cortical stainless steel microelectrode arrays in M1 and S1 in 
both hemispheres (Fig. 1.A) (Nicolelis et al., 2003). Neuronal activity was 
sampled in the arm and hand areas of M1 and S1 in the hemisphere 
contralateral to the working arm. We used four electrodes of the S1 
implants for ICMS delivery. Stimulation was applied to the right hemi- 
sphere arm area of S1 in monkey M and the right hemisphere leg area of 
S1 in monkey N using a custom-built, four-channel, current-controlled 
stimulator (Hanson et al., 2012). The monkeys were trained to move an 


image of the forearm and the hand (animated using MOTIONBUILDER; 
Autodesk) on a computer screen with a joystick (Fig. 1B). Using this 
virtual hand, they explored virtual objects presented on the screen, which 
were visually identical, yet differed in their ATs. Each trial started with the 
monkeys placing the virtual hand over a circular target that appeared at 
the center of the screen. At this time, ICMS noise of randomly chosen 
intensity started and continued until the end of the trial. Following the 
initial central hold, two visual objects appeared on the left and on the 
right of the screen, equidistant from the center. Then, the monkeys freely 
explored each of these objects to find the one associated with a periodic 
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that point. We then measured the psycho- 
metric threshold for RAT detection in the 
absence of noise for each monkey. Cur- 
rent amplitudes in the range 5-150 pA 
were tested. The detection threshold (Ge- 
scheider, 1997) was evaluated as the cur- 
rent level at which the proportion of 
correct responses was 0.75. The psycho- 
metric curve (see Fig. 3C) for monkey M 
was slightly shifted to the left compared 
with that of monkey N and therefore its 
threshold was lower (31 vs 45 pA for 
monkeys M and N, respectively). 
In subsequent sessions, ICMS noise 
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Figure 3. Stochastic facilitation via ICMS. A, B, Fraction of trials correct per session in terms of the noise amplitude level for 


monkey M (blue diamonds) and N (red squares) for suprathreshold (A) and subthreshold (B) RAT. *Significantly different from 
average performance (p < 0.05 inall cases, randomization test). C, Psychometric curves (in terms of ICMS amplitude) for monkeys 
M (blue) and N (red). Lines, Best sigmoidal fits; open circles, performance at chance level (p > 0.05, Wilcoxon rank sum test). 


ICMS pattern, which we call the rewarded artificial texture (RAT). The 
other object lacked such ICMS feedback [null artificial texture (NAT)]. 
The RAT consisted of pulse pairs (with 10 ms interpulse interval for 
monkey M and 5 ms for monkey N) delivered at a frequency of 10 Hz. 
The monkey selected one of the objects by holding the virtual hand over 
it for 2 s, and a juice reward was given if they correctly selected the RAT 
object (Fig. 1C). Each ICMS pulse was a symmetric, biphasic, charge- 
balanced current waveform delivered through a pair of electrodes im- 
planted 3 mm from each other (i.e., bipolar stimulation). For noise 
delivery, a different electrode pair was used, implanted 1 mm away from 
the first pair of electrodes (Fig. 1A). The interpulse interval of such an 
aperiodic signal was drawn from a gamma distribution with the coeffi- 
cient of variation (ratio of the standard deviation to the mean) equal to 1 
(O'Doherty et al., 2012). 

All animal procedures were performed in accordance with the Na- 
tional Research Council’s Guide for the Care and Use of Laboratory 
Animals and were approved by the Duke University Institutional Animal 
Care and Use Committee. 


Results 

Monkeys M and N were previously overtrained in a similar active 
exploration task (O’Doherty et al., 2011), so it took them just one 
and three training sessions, respectively, to achieve ~90% accu- 
racy in the two-target task of this study. No noise was presented at 


was presented throughout each trial. 
Noise amplitude for each trial was ran- 
domly selected in multiples of 15 wA, up 
to 120 A. Data with noise were collected 
for eight and nine daily sessions for mon- 
keys M and N, respectively (monkey M: 
370 + 60; monkey N: 418 = 87 trials per 
session). For these sessions, the RAT am- 
plitude was either suprathreshold (50 wA) 
or subthreshold (20 A). Representative 
trials for a subthreshold stimulus com- 
bined with different noise intensities are 
shown in Figure 2. In these examples, the 
monkey correctly detected the subthresh- 
old RAT (20 pA) in the presence of an 
optimal noise (30 wA) (Fig. 2B), while 
failing to do so when the noise amplitude 
was too low (15 wA; Fig. 2A) or too high 
(90 wA; Fig. 2C). Figure 2 also depicts the 
neuronal ensemble activity recorded in 
M1. The firing rates of M1 neurons were 
clearly modulated in association with the 
arm movements and reflected movement 
speed and direction. 

Figure 3A shows the fraction of correct 
detection (averaged by session) as a func- 
tion of noise amplitude for suprathresh- 
old RATs (50 A). Note that in the absence of noise, monkey M 
performed better (89.8% correct) than monkey N (77.9% cor- 
rect), which is consistent with a more rapid saturation of the 
psychometric curve for monkey M (Fig. 3C). For monkey M, the 
performance decayed from this high level after noise amplitude 
exceeded 30 wA [significantly lower success rate compared with 
the average for each noise level >30 wA, p < 0.05, randomization 
test (Edgington and Onghena, 2007)]. It is possible that we only 
observed decay simply because the performance was close to sat- 
uration from the start and could not improve in principle. How- 
ever, for monkey N, whose performance level had room for 
improvement, we observed a similar effect: the performance was 
resistant to weak noise and started to deteriorate when noise 
amplitude exceeded 120 wA (p < 0.001, randomization test). 

Changes in the monkey’s detection performance with noise 
addition were very different when subthreshold RATs (20 mA) 
were used. This dependency was characterized by a nonmono- 
tonic curve with peaks at noise amplitude of 60 and 45 wA for 
monkeys M and N, respectively. Thus, for monkey M, elevated 
performance of ~0.7 was observed for three noise amplitudes: 
30, 45 and 60 wA (p < 0.05 for all points, randomization test). 
For monkey N, the curve had a single peak of 0.76 at 45 wA (p< 
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0.01, randomization test). Therefore, in A 
both monkeys, the addition of a moderate 
level of ICMS noise to a weak AT stimulus 
improved the animal’s detection perfor- 
mance. Furthermore, neither the presence 
of ATs nor the addition of noise substan- 
tially affected movement-related modula- 
tions observed in M1 (Fig. 2), nor did it 
degrade the performance of a decoding al- 
gorithm that extracted the position of the 
virtual arm from the population activity 
of M1 neurons [unscented Kalman filter 
(Li et al., 2009); monkey M: 125 units, 
monkey N: 65 units; Fig. 4]. 


x-axis position (cm) 


Discussion 

The present findings demonstrate for the 
first time SF effects for an artificial tactile 
sensation. The ability of monkeys to de- 
tect weak ICMS signals was clearly im- 
proved by the addition of moderate levels 
of ICMS noise. This result is consistent B 
with previous model and experimental 
demonstrations of SF in physiological sys- 
tems (Moss et al., 2004). Interest in SF for 
neural systems has dramatically increased 
over the last 20 years, driven by theories 
claiming that noise may play an important 
role in normal neural processing (Mc- 
Donnell and Abbott, 2009; McDonnell 
and Ward, 2011). Since the presence of 
moderate amounts of internal noise may 
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be normal and even advantageous for 
neural systems, several publications pro- 
posed implementing noise and SF in arti- 
ficial systems designed to restore normal 
functions to people with disabilities, for 
example in cochlear implants (Morse et 
al., 2007), assistive devices for postural 
balance (Harry et al., 2005), and hand sen- 
sorimotor tasks (Kurita et al., 2011). Here 
we demonstrated the practicality of this idea for prosthetic sen- 
sation evoked by ICMS. By adding background noise, we effec- 
tively gained control over the detection threshold for an artificial 
tactile sensation. It has yet to be determined how SF would work 
in the case of prosthetic sensations evoked by spatiotemporal 
input patterns delivered through multielectrode implants 
(Fitzsimmons et al., 2007). For such multichannel systems 
with stimulation sites distributed over the cortical surface, the 
superposition of spatiotemporal noise patterns could enhance 
signal detection in both the temporal and spatial dimensions. 
Furthermore, SF patterns distributed across corticothalamic 
loops could be obtained in sensorized neural prosthetics that use 
thalamic stimulation as an additional channel for artificial so- 
matic sensation (Heming et al., 2010). Although in our present 
study we did not manipulate the statistics of the noise channel, it 
would be of great interest in the future to examine a variety of 
noise-frequency characteristics. If noise contains particular fre- 
quencies, SF may accordingly filter frequency components of the 
input signal. Importantly for practical neuroprosthetic systems, 
the potential risks of neural damage associated with ICMS may be 
lower for a SF-based neural prosthetic because of the lower cur- 
rents injected into the neural tissue (McCreery et al., 2010). In 
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Movement predictions from M1 activity. A, Position of the avatar and off-line predictions from M1 activity for six 
consecutive trials with different noise amplitudes. B, C, Performance of the decoding algorithm measured as the correlation 
coefficient in terms of noise level for monkey M (B) and monkey N (C). n.s., No significant difference (one-way ANOVA test, n = 
1724, p > 0.57 for monkey M; n = 3421, p > 0.07 for monkey N). 


conclusion, we suggest that the incorporation of SF effects in 
sensorized neural prosthetics may enhance a variety of functional 
capabilities. 
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Perceiving invisible light through a somatosensory 
cortical prosthesis 


Eric E. Thomson!, Rafael Carral’ & Miguel A.L. Nicolelis! 234° 


Sensory neuroprostheses show great potential for alleviating major sensory deficits. It is not 
known, however, whether such devices can augment the subject's normal perceptual range. 
Here we show that adult rats can learn to perceive otherwise invisible infrared light through a 
neuroprosthesis that couples the output of a head-mounted infrared sensor to their soma- 
tosensory cortex (S1) via intracortical microstimulation. Rats readily learn to use this new 
information source, and generate active exploratory strategies to discriminate among infrared 
signals in their environment. S1 neurons in these infrared-perceiving rats respond to both 
whisker deflection and intracortical microstimulation, suggesting that the infrared repre- 
sentation does not displace the original tactile representation. Hence, sensory cortical 
prostheses, in addition to restoring normal neurological functions, may serve to expand 
natural perceptual capabilities in mammals. 
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ost sensory neuroprosthetic systems aim to restore 

function within the same modality as a pre-existing 

pao deficit. For instance, cochlear implants restore 
auditory function!, and stimulation along the visual pathway can 
restore visual function®3, ‘ 

In some cases, however, it might be necessary to augment, 
rather than simply restore, the usual function of a sensory area*”. 
For instance, in people that suffer permanent damage to the 
visual cortex, we might ask the somatosensory cortex to take over 
some of the roles of the visual system. Such cross-modal plasticity 
has been demonstrated in juvenile animals®*. For example, 
rewiring experiments in newborn ferrets have shown that when 
visual inputs are rerouted to the auditory cortex, the auditory 
cortex a many anatomical and functional properties of the 
visual cortex”®, and even mediates visually guided behaviour’. 

Such results suggest that the function of a primary sensory 
area can dramatically change depending on the type of input 
it receives from the environment”!°. To date, however, the 
literature lacks a clear demonstration of such functional plasticity 
in normal adult mammals. This would require that the adult 
brain be plastic enough to extract novel information embedded 
within pre-existing representations, and use this information to 
generate appropriate behaviours. 

In the present study, we test whether adult rats can incorporate 
a novel sensory modality into their perceptual repertoire. 
Specifically, we examined whether adult rats could learn to 
discriminate among infrared (IR) sources after we coupled 
the output of a head-mounted IR detector to electrical 
microstimulators in the whisker representation of S1. We 
discovered that, after training with this sensory prosthetic device, 


Port 1 


IR level (v) © 


adult rats learned to navigate their IR world as if they had 
acquired a novel distal sensory modality. 


Results 

Behavioural performance. We initially trained six rats on a 
simple visual discrimination task. Rats were placed in a circular 
chamber that included three reward ports (Figs 1a,b). On each 
trial, a visible LED was activated in one of the ports, and rats were 
rewarded with water for poking their nose in that port. Once they 
reached criterion on this task (70% correct, after 25+5 days 
(mean + s.e.m.)), we surgically affixed an IR detector to the rat’s 
head, and implanted stimulating microelectrodes in the whisker 
region of S1 cortex (Figs 1c,d). 

After this surgical procedure, we returned the animals to the 
same chamber, where they had to learn to perform the same task 
using IR light, which is invisible to rats (see Methods for details 
on the task and training)!!. To allow the rats to perceive IR light 
levels, the value of the IR detector output was converted into a 
pattern of electrical stimulation in $1, with stimulation frequency 
updated every 50ms depending on the IR intensity detected 
(Fig. 1d). Electrical stimulation frequency increased as rats 
approached, or oriented their heads toward, an IR source (see 
Supplementary Movie 1). Note that such electrical stimulation in 
S1 is known to induce some type of tactile sensation in humans! 
and monkeys! 

It took 2646 days for all six implanted rats to learn to 
discriminate among the IR sources at or above the criterion used 
in the initial visual task (> 70% correct). While training with their 
new IR gear, rats underwent clear changes in behavioural strategy. 
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Figure 1 | Methods. (a) Set-up of the IR behaviour chamber. Three reward ports line the walls of the circular chamber. The proximity of the ports to one 
another is indicated by the angle 0. The rat has an IR detector (red) affixed to its head, and the cone emanating from the detector represents the area within 
which it will respond to IR stimuli. The red lines emanating from port 3 represent the IR signal emitted from that IR source. (b) Arrangement of each reward 
port, which includes a recess with a water spout, an IR LED above, and a broad-spectrum visible LED below. (¢) Design of stimulating electrodes (see 
Methods for details). The inset shows how each stimulating electrode pair is configured in the array (scale bar, 300 um). (d) Example of electrode 
placement. Cytochrome oxidase-stained barrel field shows the location of four stimulating electrodes. Each penetration is indicated with a red asterisk. 
Reference line: 500 um. (e) Coupling IR levels with ICMS. On each trial, the IR light turns on, which activates the IR detector that is mounted on the rat's 
head. Processing converts the detected IR level into a stimulation frequency. This value is sent to the microstimulator, which produces the desired current 
pulses. The inset on the right-hand-side illustrates the structure of each biphasic waveform in the pulse train. 
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At first, they did not associate intracortical microstimulation 
(ICMS) with the task, and would poke randomly in the reward 
ports, occasionally scratching their faces in response to 
microstimulation. Eventually they learned to actively forage 
through the behaviour chamber, sweeping the IR sensor on their 
heads back and forth to sample their IR world (see 
Supplementary Movies 2-4). 

Quantitatively, the rats’ performance increased from 41 +6 to 
93 +2% correct during learning (Fig. 2a), and their best single- 
session performance averaged 95 + 3% correct. Their behavioural 
latency on correct trials (the time between stimulation onset and 
poking in the correct reward port) decreased significantly as they 
improved at the task (latency dropped from 2.3+0.01 to 
1.3 +0.03 s; Fig. 2b; r= — 0.71; P= 1.9017 x 107!" (¢-test)). 

We performed a series of additional psychophysical tests on the 
new modality. First, we varied task difficulty by changing the 
angle between the ports. Moving the ports closer together 
increased uncertainty about the stimulus source, as the light from 
each IR source had a broad wavefront (Fig. la), and the IR 
detector had a relatively wide ‘receptive field’ (see Methods). 
While animals consistently performed above 90% of the trials 
correctly when the ports were 90° apart, this performance 
dropped off quickly as the angles between the IR sources were 
reduced below 60° (Fig. 2c; Supplementary Movie 3). Behavioural 
performance was significantly dependent upon task difficulty 
(P=1.1 x 10~ !°; analysis of variance). 

Because the rats swept their heads back and forth to sample the 
IR signals, we hypothesized that the rats were sensitive to the 
intensity of IR light, not just its presence or absence. To test this, 
in some sessions we randomly interspersed trials in which 
stimulation frequency was held constant, so the animal only 


received binary information about IR presence (Fig. 2d, left). 
Performance was significantly degraded in such constant- 
frequency trials (P=0.0001; two-way analysis of variance), 
demonstrating that the rats were sensitive to graded IR signals 
(Fig. 2d, right). 

Rats are typically considered blind to the IR spectrum!!: their 
cone spectral sensitivity is negligible above 650 nm!*, which is 
well outside the spectral emission of our IR source (940 nm peak 
emittance, with a range of non-zero emission between 825 and 
1000 nm; see Methods). However, in the sensory prosthetics 
literature there has been some concern about the true range of 
spectral visual sensitivity of the rat!°, so we also experimentally 
checked whether our animals succeeded by using their visual 
system to discriminate the IR cues. In two animals we added 
random ‘no stimulation’ trials in which the IR light turned on, but 
there was no stimulation delivered to the cortex. Performance on 
the task was abolished in all 11 sessions, dropping from 69 + 3.2 
to 8+ 5.2% correct, a significant change (P=7.6 x 10~ °; paired 
t-test) (Fig. 2e). Percent correct was below 33% on the trials 
without stimulation because in the majority of trials the animals 
did not poke in any of the ports, and simply let the trial time out 
(see Online Supplementary Movie 4). 


Analysis of activity in S1. Next, we investigated the effects, 
within S1, of acquiring the new sensory modality. For instance, 
are some neurons ‘hijacked’ by IR inputs!®, such that they no 
longer respond to whisker stimulation? To address this question, 
we recorded from multi-electrode arrays chronically implanted in 
S1 infragranular cortex in two well-trained rats (N= 76 single 
units over five sessions). We presented three different stimuli, 
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Figure 2 | Rats discriminate among IR sources using graded stimulation frequencies. (a) Learning curve for IR-only trials. Graph shows percentage 
of correct trials as a function of session number (130 sessions in four rats). Black circles/lines indicate mean/s.e.m. for blocks of three sessions. (b) Latency 
decreases as rats learn the task. Scatter plot of latency on IR-only trials, using same conventions as panel a (data are from 66 sessions in two rats). 
(¢) Discrimination performance varies with angle between ports. Plot shows percentage of correct trials versus task difficulty (angle between ports). Data 
are from 100 sessions in two rats. (d) Performance degrades when stimulation frequency is constant rather than variable. Plot on the right shows 
performance on trials in which the stimulation frequency is variable (top left inset) versus held constant (bottom left inset). Data are from 34 sessions in 
two rats. (e) When stimulation is turned off, performance is abolished. Left bar shows performance on trials with IR light turned on, but no stimulation. The 
right bar shows the performance from the same sessions for the trials in which we stimulated as usual. Error bars are s.e.m. Data are from 11 sessions in 


two rats. 
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each lasting 200 ms: unilateral whisker deflection (via air puffs), 
ICMS (using current magnitudes that were applied during the 
task) and both stimuli delivered simultaneously. 

S1 neurons were clearly not hijacked by ICMS, even after many 
months of microstimulation. In fact, most (83%) S1 neurons 
showed quite robust responses to whisker deflection (Fig. 3a,b). 
The breakdown of all response types is detailed in Fig. 3c. Briefly, 
of 76 units, 84% (64/76) responded to one or the other stimulus, 
while 16% (12/76) showed no significant response to either 
stimulus. The majority (83% (63/76)) showed significant 
responses to whisker deflection. Of these, 83% (52/63) were 
multimodal neurons, exhibiting significant responses to ICMS as 
well. Only one neuron in our sample showed a response to ICMS 
but not whisker deflection. Among the multimodal S1 neurons, 
the response to both stimuli delivered simultaneously was highly 
sublinear (Fig. 3d; P=4.6 x 10~ 1”; two-tailed paired t-test). 

Population responses in S1 increased according to a saturating 
function of whisker deflection magnitude (Fig. 3e). Response 
magnitudes also increased with ICMS frequency up to 350 Hz, 
but dropped off at 400 Hz, likely due to adaptation (Fig. 3f). This 
neuronal response-profile is akin to an IR receptive-field for the 
range of IR stimulation frequencies employed in the task. 


Discussion 

In 1969, Bach-y-Rita performed the classic ‘sensory substitution’ 
experiments, in which visual stimuli were projected onto the skin 
via mechanical actuators, allowing congenitally blind patients to 
experience a visual world for the first time*. Here we have applied 
the logic of sensory substitution directly to the somatosensory 
cortex, bypassing the body’s periphery, with the goal of building a 
cortical sensory prosthesis capable of augmenting the subject’s 
perceptual capability’. Instead of giving a binary, top-down 
signal instructing the rats where to move!’, we connected S1 to a 
new graded sensory cue available in their environment, and they 
spontaneously adopted novel foraging behaviours in response. 
Note that, in principle, this experimental paradigm could use any 
novel stimulus (for example, magnetic or radio waves) to be 
represented in S1. 

We observed that neurons in the stimulated regions of S1 
maintained their ability to respond to whisker deflection. This 
suggests that two different cortical representations (one tactile 
and one IR) became superimposed on the animal’s S1 cortex, 
creating a novel bimodal processing region. However, it is 
important to emphasize that behavioural studies will be needed to 
determine the consequences, for whisker-based tactile discrimi- 
nation, of adding this new information to S1. 

The mechanisms by which animals learn to use this new 
information source will also be an interesting avenue for future 
research, as such research should suggest how to accelerate 
sensory prosthetic acquisition. To investigate the mechanisms of 
plasticity, including those theories implicating glial cells!®, it will 
be helpful to examine functional and structural changes in 
cortical and subcortical tactile processing centres as rats learn to 
discriminate among IR sources. 

Overall, our behavioural results suggest that animals initially 
treated S1 electrical stimulation as an unexpected whisker 
deflection, and eventually treated it as a stimulus originating 
away from the body in the surrounding environment. However, 
we are unable, using the methods in this paper, to determine 
whether the fully trained rats consciously experienced micro- 
stimulation as a new distal sensory modality, or simply learned to 
associate a tactile sensation with an otherwise imperceptible distal 
sensory cue. This is a question that could presently be addressed 
with sensory substitution experiments in humans. Indeed, one 
such study suggests that some subjects experienced tactile stimuli 


as visual in nature after training with a visual-to-tactile peripheral 
substitution device!?. 

The next generation of sensory neuroprosthetics research in 
humans will likely continue to rely on ICMS rather than 
optogenetics”””!, Indeed, our experimental strategy has been 
quite different from optogenetic approaches*?-”°: instead of 
stimulating a specific cell-type population, we indiscriminately 
stimulated all S1 neurons in the vicinity of the stimulating 
electrodes’ tips. Despite the fact that this unusual signal was 
delivered at extremely high frequencies, and likely spread through 
most of $17°-?8, these animals readily learned to exploit this new 
sensory channel. 

One potential application of the technology used in this study 
is in the design of a new generation of motor neuroprostheses 
that, instead of simply sending a brain-derived motor control 
signal to move a prosthetic limb, would provide continuous 
sensory feedback to the user’s brain from the prosthesis??~3!. 
Such closed-loop bidirectional brain-machine-brain interaction 
could significantly improve reaction time, behavioural accuracy 
and likely aid the integration of the limb into the user’s internal 
body image?703?,33, 

However, there is another aspect of the present work that has 
rarely been explored in the neuroprosthetics literature: the 
potential to expand or augment a species’ normal perceptual 
range. In that pursuit, we have implemented, as far as we can tell, 
the first cortical neuroprosthesis capable of expanding a species’ 
perceptual repertoire to include the near IR electromagnetic 
spectrum, which is well outside the rat photoreceptors’ spectral 
sensitivity. Thus, by taking advantage of this novel paradigm, our 
rats were able to transcend the limitation of perceiving only those 
stimuli that can activate their bodies’ native sensory transducers. 


Methods 

Behavioural task and training. All experiments were performed on female Long 
Evans rats ~ 14-weeks-old (~ 250g; Harlan Laboratories). Rats were trained in a 
chamber with three reward ports that were situated 90° apart (Fig. 1a). Each port 
was fit with a visible broad-spectrum LED, and an infrared LED (Opto Semi- 
conductors Inc., 940 nm peak emittance (range of non-zero emissions was between 
825 and 1000 nm), and IR intensity dropped to half-max at 120°) (Fig. 1a). We first 
trained water-deprived rats to poke in the port whose visible LED was activated. 
Each trial began when a visible light in one randomly selected port was activated. 
Rats received water when they broke the photobeam in that port (correct trials). 
Trials counted as incorrect when the rats poked in a different port, or did not poke 
at all and let the trial terminate (this was set to occur between 15 and 20 seconds 
after the onset of the light). On incorrect trials, they received no water, an error 
tone, and a longer delay to the next trial. For some rats, we delivered air puffs to the 
face on incorrect trials. 

Once animals performed above a criterion value (> 70% correct) >4 days in a 
row, we implanted an array of stimulating electrodes into S1 (Fig. 1d). The array 
includes an IR detector attached to the connector (see Fig. 1c and Surgery below). 
After the animals recovered from surgery, we determined the minimal currents 
required to evoke a behavioural response in at least two electrode pairs (thresholds 
were between 1 and 200 1A, see Stimulating Electrodes and Stimulator below for 
details). We then trained them in the same behavioural chamber, but incrementally 
replacing visible light with ICMS linked to IR levels from their detector (Fig. le). 

Initially, each trial had the same structure as trials in the initial behavioral task 
described above, except that the onset of the visible light was preceded by IR-level 
dependent stimulation (up to 400 Hz) that lasted 0.6-1.5s. In four of six rats, we 
began with brief durations of stimulation (600-700 ms) to acclimate them to sti- 
mulation. In two of the rats, we started with longer stimulation durations to get 
estimates of behavioural latency as they learned the task (Fig. 2b). Thus the animals 
learned that stimulation indicated the presence of the visible light. The stimulation 
frequency was exponentially proportional to the IR levels (Fig. le). We used an 
exponential function because IR intensity dropped logarithmically from the IR 
source. We stimulated at high frequencies for two reasons: one, in preliminary 
experiments we found that using a lower frequency range (that is, 0-100 Hz) 
yielded less reliable performance; two, we wished to avoid evoking kindling sei- 
zures*+, Stimulation frequencies were updated every 50 ms based on IR levels, and 
it took ~5 ms for the stimulation frequency to actually change in the rat once the 
command was sent. Pulse frequency was the only variable that tracked IR levels: 
current amplitudes were kept the same for each pulse. 

Once it became clear, based on visual inspection, that the animals were 
comfortable with stimulation, we added ‘IR-only’ trials in which the IR-dependent 
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Figure 3 | Neurons in S1 respond to both whisker deflection and ICMS. (a) Sample peristimulus time histograms from neurons (rows) recorded simultaneously 
in an anesthetized animal during whisker deflections. The grey strip indicates time of whisker deflection (0-200 ms). Blue/red lines indicate mean + three s.d. values 
from the mean firing rate during the baseline period before stimulation. Rows 1-5 are single-unit responses, while row 6 is a representative multiunit response. (b) 
Sample peristimulus time histograms from the same neurons in panel a under three stimulus conditions. Left column: response to whisker deflection (38 psi air puff), 
but with the response during stimulation clipped out to enable comparison with the other two columns. Middle column: response to ICMS only (250 Hz). Right 
panel: response to simultaneous whisker deflection and ICMS. The light yellow strip indicates the time, after stimulation, we used to compare response magnitudes 
in the three cases. Note there are no responses during electrical stimulation because of electrical artifacts, so the firing rates drop to zero during those epochs. 
(€) Overall distribution of response types. Columns are segregated by whether neurons responded significantly to both stimuli, only one stimulus, or neither. 

(d) Mean (+s.e.m./ — standard deviation) response of multimodal neurons to each of the three stimuli. The observed response to both stimuli is significantly 
sublinear (fourth bar indicates expected response under assumption of linearity). Asterisks indicate significance with P<1.0 x 10~ © (paired, two-tailed t-test). 
(e) Multiunit response magnitude (mean firing rate + s.e.m.) as a function of whisker deflection magnitude (intensity of air pressure applied to whiskers) for 16 
channels in two animals. Zero is baseline (mean rate before stimulus onset), and responses for each channel were normalized to the maximum mean response over 
all three stimulus conditions. (f) Multiunit response magnitude versus ICMS frequency, with the same conventions as in panel e, with 13 channels in two animals. 
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stimulation would appear without any accompanying visible light. That is, on these 
IR-only trials they could only use the signals from ICMS to get to the correct port. 
They trained on this until their performance on the IR-only trials reached a 
criterion value of 70% correct. For some rats that stayed above criterion for four 
sessions in a row, we then varied task difficulty in which across sessions, we 
pseudo-randomly chose a new angle between the ports (either 90, 60, 45 or 30 
degrees; Fig. 2c). 

On some sessions, we added a small percentage of trials (~ 15%) in which the 
stimulation frequency was held constant, regardless of the IR intensity. Such 
constant-frequency trials allowed us to compare performance when the stimulation 
frequency depended on IR levels with those trials when stimulation frequency was 
a constant function of IR intensity (Fig. 2d, left hand side). 

All behaviour control was run in custom Matlab scripts using the data 
acquisition toolbox (run with NIDAQ PCI-7742 card, National Instruments). 


Stimulating electrodes and stimulator. In preliminary studies, we found that the 
thresholds for evoking behavioural responses were lower with electrode pairs 
within the same penetration of the cortex. Hence, for each biphasic stimulating 
electrode, we joined pairs of 30-j1m stainless steel microwires to one another, each 
pair separated by 300 1m (Fig. Ic inset). We attached a single infrared detector 
(Lite-On Inc) to the connector (Omnetics), and powered the IR detector through 
the two extra reference pins on the connector. The phototransistor in the detector 
had a peak spectral sensitivity at a wavelength of 940 nm. The range of sensitivity 
was from 860 to 1020 nm, and its ‘receptive field’ (that is, the angular range within 
which a 940 nm test stimuli would evoke a response from the detector) was 20° in 
diameter at half-max (Fig. 1a). 

We used bipolar stimulation with charge-balanced, biphasic pulse trains, using a 
custom-controlled stimulator, as described elsewhere*°. Pulses were 100 ps in 
duration, with 50 1s between the cathodic and anodic phases of the pulses (Fig. le). 
Current magnitudes varied between 1 and 300 1A. Before training rats on the IR- 
version of the task, we determined current thresholds by placing them in the empty 
behavioural chamber (Fig. la), and stimulating with 1 1A at 200 Hz for 500 ms. If 
an animal noticeably moved in response to stimulation (this typically involved 
locomoting, moving their heads to the side or scratching at their faces), that current 
level was taken as the threshold current. If not, we increased the current amplitude 
by 50%, keeping the frequency and duration the same, until we noted such a 
response. We used electrical microstimulation in lieu of optogenetics in this study, 
as one goal is to integrate this technology into human studies in the near future 
(see Discussion). 


Surgery. Detailed surgical procedures are described elsewhere*®. Briefly, we 
implanted the stimulating electrodes into $1 (— 2.5mm posterior and 5.5mm 
relative to bregma, 1.5 mm deep) under pentobarbital anaesthesia (0.065 mg g~ }), 
and the rats were given at least a week to recover from the surgery before being 
deprived again. The Duke University Institutional Animal Use Committee 
approved all surgical and behavioural methods. 


Neural recording and sorting. The basic recording set-up is described in detail 
elsewhere*®. Namely, on those channels that we were not using for stimulation, 
we recorded neural activity using the (Multichannel Acquisition Processor 
(Plexon, Inc., Dallas, Texas)). To stimulate and record simultaneously without 
damaging the head stage, we peeled off the wires connected to the stimulator from 
the connector to bypass the head stage, while the other channels went directly to 
the head stage. Sorting of neural data is also described elsewhere*®. Briefly, in 
addition to template-based online sorting, all voltage traces around a threshold- 
crossing event were saved for offline sorting. For offline sorting, we used clustering 
in principal component space, signal to noise ratio and autocorrelation functions 
that showed a clear absolute and relative refractory period to determine whether 
the data came from single units or multiple units. 

In our recordings, we lightly anesthetized head-fixed animals under isofluorane 
anaesthesia (0.8-3% isofluorane mixed with pure oxygen). We controlled the 
duration and magnitude (psi) of air via a modified fluid dispenser (Oki DX-250, 
Garden Grove, CA), which was controlled with TTL pulses sent from Matlab. 
Before recording, we positioned the air dispenser to stimulate the majority of the 
large whiskers, ~ 10mm from the whisker pad. We delivered three different 
stimuli, in pseudo-random order: air puff to whiskers, electrical stimulation and 
both stimuli delivered simultaneously (see Fig. 3). 


Statistical methods. We calculated significant responses in peristimulus time 
histograms using a bootstrap cumulative sum algorithm described in more detail 
elsewhere*’. This involved taking bootstrap samples of the cumulative response 
during a baseline period (period before time zero in Fig. 3a), and using these to 
generate a cumulative sum 95% confidence interval (with Bonferonni correction for 
multiple comparisons), and comparing this to the actual cumulative sum during 
the period of stimulation. A significant response was when the actual cumulative 
sum left the 95% confidence interval, and ended when the response dropped below 


mean firing rates during the 130 ms period following the offset of both stimuli. To 
compare relative response magnitudes among multiple neurons, we normalized the 
mean response to each stimulus by the maximum mean response over all three 
stimulus conditions, so the maximum response was assigned a value of one. 

To generate the predicted responses to both stimuli delivered simultaneously 
(Fig. 3c), we used the normalized responses described in the previous paragraph. 
For a linear system, the predicted response to both stimuli would simply 
be the sum of the responses to each stimulus taken individually. That is, 

R(E+ W) = R(E) + R(W), where E is electrical stimulation, W is whisker deflection, 
and R(x) signifies the response to input x. 
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Can the adult brain assimilate a novel, topographically organized, sensory modality into its perceptual repertoire? To test this, we 
implemented a microstimulation-based neuroprosthesis that rats used to discriminate among infrared (IR) light sources. This system 
continuously relayed information from four IR sensors that were distributed to provide a panoramic view of IR sources, into primary 
somatosensory cortex (S1). Rats learned to discriminate the location of IR sources in <4 d. Animals in which IR information was 
delivered in spatial register with whisker topography learned the task more quickly. Further, in animals that had learned to use the 
prosthesis, altering the topographic mapping from IR sensor to stimulating electrode had immediate deleterious effects on discrimina- 
tion performance. Multielectrode recordings revealed that $1 neurons had multimodal (tactile/IR) receptive fields, with clear preferences 
for those stimuli most likely to be delivered during the task. Neuronal populations predicted, with high accuracy, which stimulation 
pattern was present in small (75 ms) time windows. Surprisingly, when identical microstimulation patterns were delivered during an 
unrelated task, cortical activity in $1 was strongly suppressed. Overall, these results show that the adult mammalian neocortex can readily 
absorb completely new information sources into its representational repertoire, and use this information in the production of adaptive 
behaviors. 


Key words: barrel cortex; rat; sensory prosthetic; whisker system 


Significance Statement 


Understanding the potential for plasticity in the adult brain is a key goal for basic neuroscience and modern rehabilitative 
medicine. Our study examines one dimension of this challenge: how malleable is sensory processing in adult mammals? We 
implemented a panoramic infrared (IR) sensory prosthetic system in rats; it consisted of four IR sensors equally spaced around the 


circumference of the head of the rat. Each sensor was coupled to a microstimulating electrode placed in the somatosensory cortex 
of the rat. Within days, rats learned to use the prosthesis to track down items associated with IR light in their environment. This is 
quite promising clinically, as the largest demand for sensory prosthetic devices is in adults whose brains are already fully 
developed. 


patients regain functional levels of sensory processing (Macherey 
and Carlyon, 2014). In the future, these peripheral prosthetic 
devices will likely be complemented by systems with higher 
throughput that directly activate neurons in the CNS, in partic- 
ular the cerebral cortex (Dobelle and Mladejovsky, 1974; Dobelle, 


Introduction 
In recent decades, sensory prosthetic systems such as cochlear 
implants (Wilson et al., 1991) have helped tens of thousands of 
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2000; Coulombe et al., 2007; Srivastava et al., 2009; Davis et al., 
2012; O'Doherty et al., 2012; Andersen et al., 2014; Lewis et al., 
2015). Such central neuroprosthetic systems could eventually be- 
come as routine in the treatment of sensory deficits as deep brain 
stimulation has become in the treatment of Parkinson’s disease 
(Deuschl et al., 2006; Fuentes et al., 2009; Santana et al., 2014; 
Yadav et al., 2014) and even some psychiatric disorders such as 
depression (Mayberg et al., 2005). 

To work properly, cortical sensory neuroprosthetics will re- 
quire that the adult brain be plastic enough to continuously pro- 
cess real-time streams of synthetic information sources, and then 
use the extracted information to guide appropriate behavioral 
responses. Classically, the highest levels of cortical plasticity have 
been observed during early development (Frost and Metin, 
1985). For instance, in a set of rewiring experiments in newborn 
ferrets, primary auditory cortex was induced to process visual 
information when visual information was routed to Al (Sur et al., 
1988). Following this procedure, individual neurons in rewired 
Al exhibited visual response properties typically associated with 
neurons in V1, such as center-surround receptive fields (RFs; 
Frost and Metin, 1985; Sur et al., 1988; Horng and Sur, 2006), and 
rewired Al-mediated visual behaviors in these transformed ani- 
mals (von Melchner et al., 2000). 

Recently, we reported similar findings in adult rats in which 
information from a single infrared (IR) sensor was delivered, via 
cortical microstimulation, to primary somatosensory cortex (S1; 
Thomson et al., 2013). After a month of training with this hard- 
ware, rats readily discriminated the location of IR sources in the 
environment in exchange for a water reward. This cortical pros- 
thesis completely bypassed the native sensory transducers of the 
animals, delivering sensory information never experienced by 
these animals into a region of cortex already devoted to process- 
ing somatosensory information. This type of sensory enrichment 
system was recently extended when the output of a geomagnetic 
compass was coupled to stimulating electrodes in S1 of rats 
placed in a T-maze (Norimoto and Ikegaya, 2015). 

Overall, these experiments show that the adult mammalian 
brain can adaptively absorb completely new sources of sensory 
information delivered directly to cortical tissue. They also dem- 
onstrate the viability of using rodent models in the development 
of cortical sensory prostheses. However, several questions left 
open by these initial studies will have to be addressed for the 
development of more realistic neuroprosthetic systems. For in- 
stance, how does the brain respond to these qualitatively new 
information sources? Will animal performance improve with the 
addition of more sensors, or would this additional information 
be a source of confusion? Does the spatial distribution of the 
stimulating electrodes matter? If different combinations of sen- 
sors were inactivated, would behavioral performance degrade 
gracefully, the way natural sensory systems do? 

Many of these questions would be impossible to ask using a 
neuroprosthesis containing only a single transducer. Thus, in the 
present study we have extended the IR discrimination paradigm 
to include four IR sensors distributed to provide a full panoramic 
(360°) representation of the surrounding IR environment (Fig. 
1A). The rats were not confused by the additional sensors and 
stimulating electrodes, but learned the task an order of magni- 
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tude faster than when equipped with just a single IR sensor. In 
animals trained to discriminate IR sources, individual $1 neurons 
exhibited both IR and robust tactile evoked responses. These and 
many other neurophysiological adaptations described below un- 
derlie the emergence of a multimodal IR tactile representation in 
the adult rat $1, allowing animals to perform an IR discrimina- 
tion task at nearly 100% accuracy. 


Materials and Methods 


Training and behavioral task. All experiments were conducted on female 
Long-Evans rats ~10 weeks old (200-224 g; Harlan Sprague Dawley). 
For behavioral training, rats were trained in a circular chamber (50.8 cm 
diameter) with four reward ports that were situated 90° apart. The reward 
was given through a poke hole that was fit with a photo beam. Each port 
was additionally fit with a visible broad-spectrum LED, and an infrared 
LED (Opto Semiconductors; 940 nm peak emittance; range of nonzero 
emissions, 825-1000 nm). The IR sources had an angular width at half- 
maximum of 120°). A custom-made pushbutton was affixed to the floor 
in the center of the chamber, which rats pressed to initiate each trial (27 
mm diameter; Fig. 1A). 

Initially, 17 rats were pretrained in a simple visual discrimination task, 
in which they were placed in a circular chamber with four water ports that 
contained, in a vertical arrangement, a visible-light LED, an infrared 
LED, and a water spout (Fig. 1A). In the initial pretraining, rats learned a 
very simple visual discrimination task in which one of the four visible 
lights turned on, and animals were rewarded when they selected that 
port. Selection was indicated by breaking the photobeam in front of the 
water spot. 

Once the performance of the animals crossed 85% correct on the 
visible light (see Materials and Methods) version of the task (which took 
9 + 3d), we then prepared them for the IR discrimination training. Once 
animals learned to use the center button to start a trial, they normally 
performed well above 95% correct. First, four groups of eight microelec- 
trodes were chronically implanted in the whisker representation— barrel 
cortex— of both S1s of each rat (see Stimulating electrode implantation 
surgery). Each microelectrode group was associated with a different IR 
detector (Fig. 1B). We chose this approach as the most natural way to 
associate each IR detector with a topographic homolog in the rat $1. So, 
for instance, the front-right IR detector was intended to stimulate the 
region of S1 corresponding to the right-front whiskers, corresponding to 
the left anterior barrel field in S1. 

Next, we retrained rats on the IR version of the original discrimination 
task, replacing visible light with IR light. Thus, for rats to be rewarded in 
a trial they now had to track the IR beam all the way to the port from 
which the IR beam was being emitted. For each session, the four IR 
detectors were snapped into place on the head of each animal via a 
magnetic seal. The output of each IR detector was independently con- 
nected to the activity in the stimulation channel in the appropriate quad- 
rant of S1, so, for instance, if the head of the animal was oriented so that 
the right-front IR detector faced the active IR source, then the left-front 
region of S1 would be highly activated (Figs. 1C—E). The frequency of 
stimulation in each stimulating channel varied approximately exponen- 
tially with IR levels in its corresponding IR detector (Thomson et al., 
2013). We excluded two animals from subsequent analysis: one devel- 
oped an infection at the site of implantation, and the acrylic head cap 
came off the other after surgery. 

Initial current levels for microstimulation were found by stimulating 
in the lightly anesthetized animal a week after surgery. For each of the 
four locations, we determined the minimal threshold for at least one pair 
of stimulating electrodes, defined as the smallest current we could deliver 
(20 pulses at 200 Hz) and evoke a clear response in S1 (as determined by 
listening to responses in an audio monitor). 

We then trained animals in the same behavioral chamber, but replaced 
the visible light with intracortical microstimulation (ICMS) propor- 
tional to the level of IR light detected by a sensor. A trial was counted 
correct when the rat poked the port with the activated IR light. After the 
animals were able to perform the task solely with IR light and met the 
criterion of reaching 85% correct, we sometimes conducted a series of 
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Methods for rat IR discrimination training. A, Schematic of the behavioral chamber used for the task. Four ports are arranged symmetrically around the inner surface of a large (24 inch) 


cylinder. Each port contains a nose poke, an IR light, and a visible light. In the middle of the floor is a button the rat pushes to initiate a trial. B, Topographical organization of information from four 
IR detectors in rat S1. The IR sensors were placed 90° apart from each other, with each sensor coupled to a different stimulating electrode pair in S1. Note that the representation of the left whiskers 
is found in the right hemisphere, but the front-facing whiskers are still represented on the anterior part of the barrel field in $1 (N, through N, represent the four locations of neuronal stimulation). 
C Stimulation frequency depended on IR intensity in each sensor. The intensity of each IR light was translated into different stimulation frequencies, in real time, in its corresponding stimulation 
channel. D, Polar plot showing the response of each IR sensor as a function of angle in the chamber, when the sensor array is at a fixed position in the chamber, relative to a single activated IR source. 
The red point on the edge indicates the relative location of the IR source. E, Full-width at half-maximum (FWHM) of the response profiles as a function of position in the chamber. The red point is the 
position of the active IR source, while the FWHM is the mean FWHM ofall four sensors at the given position (see D). As you move further way, or oblique, to the source, the response profiles become 


narrower. The black point indicates the position of the data represented in D. 


additional experiments, which are described below (e.g., remapping 
experiments). 

All behavioral and microstimulator controls were performed using 
custom MATLAB scripts using the data acquisition toolbox (run with 
NIDAQ PCI-7742 card, National Instruments). 

Stimulating electrode implantation surgery. We implanted the elec- 
trodes bilaterally in $1 under full anesthesia. To initiate the anesthesia, 
rats were first put to sleep with isoflurane (Isothesia, Henry Schein Ani- 
mal Health). For final anesthesia, we used a ketamine (Ketaset, Fort 
Dodge)-xylazine (AnaSed, Akorn Animal Health) anesthetic with 100 
mg/kg ketamine and 0.06 mg/kg xylazine. During surgery, supplemental 
doses (33% of the initial dose, administered intramuscularly) of ket- 
amine were provided when necessary. Throughout surgery, we delivered 
at least 3 ml of saline (Hospira) every 2-3 h, injected subcutaneously. 
Further, 0.02 mg/kg atropine (Atropine, West-Ward Pharmaceuticals) 
was given within the first hour and then every 2-3 h until the animal 
regained consciousness. Surgeries took between 3.5 and 6 h, excluding 
recovery from anesthesia. 

Coordinates for the craniotomy were taken with the stereotactic tools 
relative to bregma at 2.5 mm posterior and 5.5 mm bilateral. Electrodes 
were lowered 1.5 mm into the brain at an angle of 10°. Electrode pairs in 
one penetration were 300 um apart, so the electrode tips of each pair were 
positioned at 1.5 and 1.2 mm below the surface of the cortex (Thomson 
et al., 2013). The craniotomy was covered with eye ointment before the 
electrodes were fixed in place, and the craniotomy was sealed using dental 
cement (Perm Reline, Coltene/Hobbylinc). 


After surgery, the rats were given at least 7 d to recover. During this 
time, we provided them with Tylenol for pain relief and monitored their 
weight. The Duke University Institutional Animal Care and Use Com- 
mittee approved all surgical and behavioral methods. 

Four IR sensor neuroprosthetic. The electrodes, implanted in the so- 
matosensory cortex, formed the first part of the neuroprosthetic. Elec- 
trodes were implanted in each hemisphere (Fig. 2B). In both 
hemispheres, the layout was identical and consisted of 16 electrodes and 
one ground electrode. Preliminary studies revealed that the thresholds 
for evoking behavioral responses were lower when two electrodes were 
paired in the same penetration of the cortex (Thomson et al., 2013). For 
our stimulating and recording electrodes, we therefore joined two 42 um 
stainless steel polyimide insulated microwires. 

The second part of the neuroprostheses consisted of an attachable cap 
that included four IR sensors. The phototransistor in the IR sensors 
(Lite-On) had a peak spectral sensitivity at a wavelength of 940 xm with 
a range of sensitivity of 860-1020 um. The sensor had a 20° width at half 
of its maximum sensitivity. The four infrared sensors were placed 90° 
apart from each other ona horizontal plane (Fig. 1B). The head cap could 
be plugged into the connector (Omnetics Connector Corporation) on 
the head of the rat and was sealed in place with a small magnet. 

To more fully characterize the response profile of the system, the IR 
sensor response as a function of IR sensor angle and position in the 
chamber was measured off-line in the behavioral chamber. This was 
done by affixing the panoramic IR sensor array onto a stepper motor 
(Portescap) that was rotated in 15° increments, with the response of the 
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Figure 2. Learning to use distributed IR information to discriminate IR sources. A, The mean percentage of correct responses for the first sessions from eight animals (mean + SEM). The dotted 
line marks the 85% threshold. Also, see Movie 1. B, The latency of the response (time between stimulation onset and animal breaking a photobeam in one of the ports) over different sessions, from 
the same rats as in A. C, Comparison of rats with one IR sensor (gray) and rats with four IR sensors (black) implanted on the head. The black line is the same data shown in A. The one sensor group 
consists of three animals and is averaged with a sliding window over 3 d. D, The number of sessions to reach 85% correct was significantly different for one vs four IR detectors. Rats with four IR 
sensors needed 3.75 + 0.45 sessions to reach 85% correct, while rats with only one IR sensor needed 22.3 + 7.62 sessions. A two-sided t test shows a significant difference between both groups 
(p = 0.0019). E, F, Circular histogram of head direction at trial onset relative to the last visited port. E, The head direction before they learned to use the IR implant. F, The head direction after they 
learned to use the IR implant. Ports are located at 0°, 90°, 180°, and 270°. Before they learned, they faced 178° away of the last visited port, and after they learned they faced 155° away from that 
port. G, Performance vs task difficulty. The schematics above show the layout of the task chamber for the four-IR sensor implant. Rats with four IR detectors got >85% correct even when the ports 
were 30° apart. The figure compares the performance of rats with four IR sensors (N = 6; red line) vs one IR sensor (N = 3; green line). The graph shows the mean percentage of correct responses, 
and the error bars show the SEM. Rats with four IR sensors showed a trend of decrease in performance in IR-only trials for smaller angles (p = 0.132, ANOVA). The performance of rats with only one 
sensor dropped significantly for 45° and 30° ( p = 0.0008 and p < 0.0001, respectively, multiple-comparison test). For 45° and 30°, rats with four IR sensors performed significantly better than rats 
with only one IR sensor (p < 0.0001 and p = 0.0029 respectively, multiple-comparison test). H, The performance decreased significantly as more sensors were deactivated (p < 0.0001, one-way 
ANOVA). The graph shows the mean/SEM of behavioral performance, the dotted line is chance performance (25%). Data are pooled from six animals, disregarding which channels were deactivated. 
The performance with only one of the four sensors was at 52.8 + 2.07% (percentage correct + SEM) and therefore was still above the chance level of 25%, regardless of which sensor was still active. 
When all sensors were deactivated, the performance dropped to 14.5 + 3.77%. A post hoc test showed that, with the deactivation of each sensor, the performance decreased significantly. J, 
Behavioral performance when specific channel combinations were inactivated shows that certain channels show a stronger influence on the performance than others. Inactivating of single sensors 
showed a trend of reduced performance but was not significant. Surprisingly, when both front-facing sensors were deactivated, the drop in performance was not significant (p = 0.363, 
multiple-comparison test). In contrast, a significant decrease occurred when both back, both left, or both right sensors were deactivated (all p < 0.0001 when compared with all sensors active 
condition, multiple-comparison test). No difference was found between the left and right sensors (p > 0.99, multiple-comparison test). When three of four sensors have been deactivated, the 
performance dropped significantly (all p < 0.001). For comparison of p values across all ablation conditions, see Table 1. 
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IR prosthesis 
movie 


Movie1. Panoramic IR discrimination, related to Figure 2. A rat locates the source of 
the IR light with the neuroprosthetic. The movie is captured with an IR-sensitive camera. 
Each trial is initiated when the rat presses the center button and a choice is made by 
poking the nose into the correct port. Audio signal is a direct feed from the microstimula- 
tor, and is not available to the rat. 


IR sensors measured at each angle to yield a full IR response profile at a 
given position in the behavioral chamber (Fig. 1D). Such measurements 
were made at each position in the behavioral chamber, at 1 inch spatial 
resolution. The main differences in responses at each position were the 
angular preference, and width at half maximum of the response. The 
latter is shown as a function of position in the chamber (Fig. 1E). 

The in-house-made stimulator used a bipolar stimulation with 
charge-balanced, biphasic pulse trains (Hanson et al., 2012). The fre- 
quency of stimulation depended on the intensity of the detected IR light. 
We used seven different frequencies, ranging from 0 to 425 Hz (Thomson 
et al., 2013). 

Neural recording. The basic setup for recording neuronal activity has 
been described in detail previously (Wiest et al., 2008; Thomson et al., 
2013). Briefly, in the channels we did not use for stimulation, we re- 
corded neural activity using the Multichannel Acquisition Processor 
MAP; Plexon). To stimulate and record simultaneously, the wires con- 
nected to the stimulator bypassed the head stage, while the other chan- 
nels went directly to the head stage. Sorting of neural data is also 
described previously (Wiest et al., 2008). Briefly, in addition to template- 
based on-line sorting, all voltage traces around a threshold-crossing 
event were saved for off-line sorting. For off-line sorting, we used clus- 
tering in principal component space and refractory periods to assign data 
either as single units or multiunits. 

Sensor ablation. To confirm that the animals used all four IR sensors 
and not only a subset of them, in six rats we ran whole sessions during 
which we deactivated the ICMS in one or multiple locations. The number 
of ICMS locations that were deactivated ranged from zero (all active) to 
four (all deactivated). We used different combinations of deactivated 
channels to test for preferences of a side or an orientation. The deactiva- 
tion of all four ICMS sites was a necessary control to test whether the rats 
relied solely on the stimulation itself or whether they used other cues to 
guide their behavior. We used three to five different ablations in one 
session, with at least 40 trials per ablation; the order in which the abla- 
tions were presented was randomly determined. A session in this exper- 
iment consisted of 140-220 trials, depending on the number of different 
ablations used in the session, 20% of which were trials with visible light. 

Varying angles between reward ports. As a common measurement of a 
sense (modality), we were interested in the spatial resolution of the ability 
of the rats to discriminate IR sources. We moved the reward ports closer 
together so that the angle between them within a session was 90°, 60°, 45°, 
or 30°. A session consisted of 60 trials at two or three angles, with 20% of 
trials with visible light. The order in which the angles were presented was 
randomly determined. This was done with six rats. 

IR sensor and S1 remapping. To test the ability to adapt to a new input 
structure, and determine whether the rats were using the spatially distrib- 
uted information provided by the prosthesis, we changed the connection 
between the four sensors and the four ICMS sites. We used three different 
permutations: front-back flip, where the front sensors controlled the 
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stimulation in the posterior ICMS sites and vice versa; left-right flip, 
where the left sensors stimulated the left hemisphere and vice versa; and 
a diagonal flip, a combination of the left-right flip and the front-back flip. 
In the diagonal flip, the front right sensor controlled the ICMS in the 
posterior barrel cortex in the right hemisphere, and all other channels 
were connected respectively. One remapping was applied consistently 
until the rats learned to use the new mapping. The threshold of learning 
was a performance of >85% correct in IR-only trials. In these experi- 
ments, to start, 100% of the trials had only the IR light as a cue. If the rats 
did not reach a performance of 25% after three sessions, we included 
trials with visible light for a few sessions. This was necessary only when 
the very first remapping was a diagonal remapping. 

Task-irrelevant microstimulation control animals. To control for 
stimulation-induced plasticity, we trained and implanted two other an- 
imals that received stimulation that was not informative for any task. 
They were treated in the exact same way as the animals in the IR discrim- 
ination task, except they performed only the visible version of the task. At 
random times during this task, they were presented with stimulation 
patterns collected from rats that performed the actual task. 

Histology. At the end of the experiments, seven rats were killed to 
confirm the electrode positions. The rats were anesthetized with 1 ml of 
Euthasol and perfused with saline, followed by a 3% paraformaldehyde 
in 0.1 M phosphate buffer. The brain was removed from the skull, and the 
subcortical structures were carved out. The cortex was carefully flattened 
for 5hina3% paraformaldehyde solution. After the cortex was flattened, 
it was fully fixed in a 4% paraformaldehyde solution overnight and then 
dehydrated in a 30% sucrose solution for another 24 h. Cryostat sections 
of 40 xm were stained with cytochrome oxidase to show the barrel fields, 
as described previously (Thomson et al., 2013). 

Analysis. Data acquisition and analysis was performed in MATLAB 
using custom-written code, the Statistics Toolbox, and the Circular Sta- 
tistics Toolbox (Berens, 2009). Here we discuss specific techniques we 
used. 

The stimulus population vector P in response to the microstimulation 
pattern S = <f,, f, fy, f,> is as follows: 


4 
P(S) = dv 


where v; is a vector pointing in the direction of IR detector i (e.g., the top 
right detector location is represented by <1, 1>), and f; is the frequency 
of stimulation in stimulating channel i. 

We used a bootstrap analysis to generate the 95% confidence ellipses 
around the mean of two-dimensional (2D) quantities. Namely, we took 
10,000 bootstrap samples from the set of all values of the quantity, and 
calculated the mean of each bootstrap sample. We then fit the distribu- 
tion of such bootstraps with a 2D Gaussian with the covariance matrix of 
the bootstrap samples, oriented in the direction of the eigenvectors, with 
the major and minor axes that determine the size of the ellipse deter- 
mined by the covariance matrix. 

Multiple-comparison tests were performed in MATLAB to determine 
individual differences when a general ANOVA was significant. The test 
consisted of multiple t tests with Bonferroni correction for multiple 
comparisons. 

In calculating the IR-RFs, we took the mean response for each stimulus 
delivered, used linear interpolation to fill in gaps for where there were no 
stimuli presented, and then smoothed the resulting map using a 2D 
Gaussian filter with a width = 30 and o = 10. 

To extract head direction and initial movement at trial onset, manual 
video analysis was used. Head bearing was determined by the nose direc- 
tion with a graduator on the monitor screen. For the initial movement, 
only trials where the IR light came from the left or right side of the animal 
head were used (45—135° or 225-315° relative to head bearing of 0°). A 
trial was counted as having head movement if the animal swiveled its 
head >10° after IR source onset; trials without such angular head move- 
ments (16% of trials) were discarded. 
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Table 1. The p values for comparison of performance in ablation experiments 
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Allon Ch1 off Ch2 off Ch3 off Ch4 off Front off Back off Right off Left off Chl on Ch2 on Ch3 on 
Ch1 off 0.999 
Ch2 off 0.895 0.999 
Ch3 off 0.137 0.803 0.998 
Ch4 off 0.051 0.609 0.985 1 
Front off 0.692 0.995 1 0.999 0.999 
Back off = * 0.007 0.050 a 0.022 
Right off * 0.097 - 0.011 0.026 = 0.998 
Left off = e 0.014 . = 0.064 0.994 1 
Ch1on * * # a * ” 0.796 0.136 0.040 
Ch2on * . = * i ee 0.027 be 0.132 0.931 
Ch3 on * * * - * . 0.038 = 0.241 0.956 1 
Ch4on x i - = : . 0.119 0.003 = 0.997 1 1 
The p values are calculated using a multiple-comparison test. Ch, Channel. 
*Values <0.001. 
Results provided them with a more complete perspective on the IR 


Behavioral task 

We trained 15 animals to discriminate the location of an IR 
source (see Experimental procedures). Briefly, each animal was 
trained in a location-discrimination task in a cylindrical arena 
that contained four water ports affixed to its border; in addition 
to a water spout, each port contained a visible and an IR light (Fig. 
1A; Thomson et al., 2013). In the task, a light in a single port 
turned on, and, to receive a water reward, the animal had to break 
a photobeam in that port. Initially each rat performed the task 
with visible light only. Once they reached 85% correct, they were 
implanted with four groups of stimulating electrodes in four 
quadrants of S1 (Fig. 1; Experimental procedures). In subsequent 
sessions, we used the output of four IR sensors, which were evenly 
distributed around the head of the rat to control the stimulation 
frequency of stimulating electrodes in four quadrants of S1 (Fig. 
1B; Experimental procedures). When one of the IR sensors was 
closer to, or oriented toward, an IR source, it evoked higher- 
frequency microstimulation in its corresponding S1 stimulating 
channel (Fig. 1C—E). 


Performance in IR discrimination task 

Thirteen animals quickly learned to discriminate IR sources us- 
ing the synthetic panoramic IR signals delivered to $1. Figure 2A 
shows the learning curve of these animals, in which the percent- 
age correct is plotted as a function of the number of days using the 
IR neuroprosthesis. On average, the animals surpassed the 85% 
correct mark within 3.75 + 0.45 sessions (mean + SEM). The 
mean best performance (the mean of their best five sessions) of 
the rats was 98.0 + 0.51% correct. As rats learned the task, the 
latency, or time between ICMS onset and selection of the port, 
decreased significantly from 5 to <2 s (Fig. 2B). Figure 2, C and 
D, compares the learning rates of rats when trained with four IR 
sensors, with the data from a previous study in which animals 
were equipped only with a single IR sensor (Thomson et al., 
2013). The rats wearing the new panoramic IR neuroprosthesis 
crossed the 85% correct threshold significantly faster than ani- 
mals implanted with a single IR detector (3.75 + 0.45 sessions 
with four IR sensors vs 22.3 + 7.62 sessions for rats with one IR 
sensor). 

Unlike rats implanted with a single IR detector, animals im- 
planted with our panoramic IR neuroprosthesis did not sweep 
their heads laterally in an attempt to sample IR signals from the 
surrounding environment (Thomson et al., 2013). Instead, once 
they received an ICMS with IR light onset, these rats often turned 
directly toward the correct port, likely because the four IR sensors 


sources (see Discussion). 

To investigate the animals’ behavior when using the new im- 
plant, we recorded the head direction when the animals initiated 
each trial. Specifically, we calculated the head bearing relative to 
the last poked port. Here we found that before the animals used 
the IR implant, they typically faced directly away from the previ- 
ous port poked (178° away). Interestingly, after they learned to 
use the implant, they faced 155° away (Fig. 2E,F). This change 
with learning was significant (p < 0.0001, nonparametric 
ANOVA), and both distributions were significantly different 
from a uniform distribution (p < 0.0001 and p < 0.0001, respec- 
tively, Raleigh test). 

In the next set of experiments, we varied the task difficulty by 
moving the IR sources closer together in the chamber, randomly 
setting the angles between ports to 90°, 60°, 45°, or 30° (Fig. 2G, 
inset, arena layout icons). Surprisingly, while we observed a slight 
decrease in performance as the ports were closer together, this 
was not statistically significant in rats wearing a four-IR detector 
neuroprosthesis (Fig. 2G; p = 0.132, ANOVA). This is in contrast 
to the results in our previous study with a single sensor (Fig. 2G; 
Thomson et al., 2013, their Figs. 1, 2). 


Ablating IR sensors 

How would the animals respond if we shut off different combi- 
nations of IR sensors, so that rats did not have the full panoramic 
view of the IR environment? We examined this in six animals, in 
which different combinations of IR sensors were inactivated on 
randomly selected trials (see Experimental procedures). We call 
such inactivations sensor ablations. 

On average, with each additional sensor ablated, the performance 
of the rats dropped significantly ( p < 0.0001, one-way ANOVA; Fig. 
2H). The p values for the comparisons of performance in all ablation 
conditions are indicated in Table 1. With the inactivation of one 
channel, performance dropped significantly when compared with 
performance when all IR sensors were active (91.4 + 0.80% correct 
with all sensors vs 86.1 + 0.95% with only three sensors active; p = 
0.0209, multiple-comparison test). With each additional channel 
deactivated, the performance decreased significantly (all p < 0.0001, 
multiple-comparison test). The performance reached 74.7 + 1.7% 
with two sensors deactivated, and 52.8 + 2.07% with three channels 
deactivated. The rats were still able to perform the task above chance 
level (25%), with only one of four IR sensors activated. When all four 
sensors were deactivated, so that the animal was receiving no infor- 
mation from the prosthetic, the performance dropped drastically, 
down to 14.5 + 3.8%, which is below chance (Fig. 2H). Note that 
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performance can be lower than chance when rats do not make a 
behavioral selection, and allow the trial to time out. 

Figure 22 summarizes the performance for every particular 
sensor ablation experiment, showing that some IR sensor combi- 
nations were more important than others for task performance. 
Surprisingly, we observed that ablating individual back-facing IR 
sensors caused a larger decrement in performance than the small 
but insignificant decrease observed when a single front-facing 
sensor was inactivated (Fig. 21). Similarly, ablating both rear- 
facing channels led to a decrease in performance to 70.0 + 2.1% 
(p < 0.0001, multiple-comparison test), which was significantly 
larger than the decrement observed after ablating both front- 
facing channels (p = 0.002, multiple-comparison test; Table 1). 
With just a single sensor active, even if it was a front sensor, 
performance was still significantly above chance (p < 0.05; see 
Discussion). 

Finally, when the two sensors on one side of the animal were 
ablated, the performance dropped significantly (to 70.1 + 3.6% 
when the right side was ablated and to 65.5 + 5.3% when the left 
side was ablated; p < 0.0001, multiple-comparison test). There 
was no significant difference between the left-only and right-only 
conditions (p > 0.999, multiple-comparison test). 


Topography remapping 

Our sensory neuroprosthetic system gives us complete control 
over the mapping between the IR sensor and the stimulating 
electrode. This allowed us to examine the rat’s sensitivity to the 
spatial distribution of IR signals within S1. More specifically, 
once animals had learned the task with the normal configuration, 
how did they react when that mapping between the IR sensor and 
stimulating electrode was changed (Fig. 3A)? 

We examined the following three topographic remappings in 
nine animals: left-right remapping, in which we connected IR 
sensors to stimulating channels in the ipsilateral cortical hemi- 
spheres instead of the usual contralateral loci (Fig. 3A); front- 
back remappings, in which the front IR sensors project to the 
barrels that represent the rear whiskers and vice versa (Fig. 3A); 
and third, diagonal remappings combine the first two remap- 
pings, completely reversing the usual topography in both dimen- 
sions. For convenience, hereafter we will call the first two 
examples—left-right and front-back—one-dimensional (1D) 
remappings, whereas the diagonal remapping is classified as a 2D 
remapping. 

The impact of all remappings was immediate and pro- 
found. For instance, in left-right remappings, when an IR 
source appeared on the right side of the animal, the rat would 
promptly move to the left and select the incorrect port. De- 
spite such understandable miscalculations, rats quickly started 
to recalibrate their performance during their first sessions 
with 1D remappings. 

Quantitatively, Figure 3B shows the behavioral performance 
for a series of remappings in one animal. It started with a front- 
back remapping (its performance the day before remapping is 
indicated by the green dot). The performance of the animal 
dropped during the first session, but it soon recovered as it 
learned the new relationship between IR sensors and stimulating 
electrodes. After animals successfully relearned the task following 
an initial remapping, we then tested their ability to adapt to sub- 
sequent remappings. In this case, after the front-back shuffle, we 
then remapped the left and right IR sources, resulting in a 2D 
diagonal remapping. Again, the animal learned the correct asso- 
ciation within a few sessions (Fig. 3B). The performance of the 
animal on subsequent remappings is indicated in Figure 3B. 
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Curiously, none of the three animals that started with a diag- 
onal remapping learned the task unaided: we used visual cues to 
help them properly recalibrate their maps. Figure 3C shows a 
representative animal that started on the diagonal remapping. 
The rat struggled on the first three sessions with below-chance 
performance. Indeed, all animals stayed at below chance level 
(25%) when first introduced to the 2D remapping, and ulti- 
mately lost motivation in the task. To keep them motivated, we 
supplemented ICMS with visible light to guide them to the cor- 
rect port (visible trials: sessions in which we used visible trials are 
indicated in Fig. 3C). Over the next few sessions, we included a 
decreasing proportion of visible trials. Such visually guided reca- 
libration of the mapping was never necessary for 1D remappings, 
but it was needed for all three animals to learn the 2D remapping. 

Figure 3, D and E, shows the pooled data for each remap- 
ping group. No difference was found between the two 1D remap- 
pings (p = 0.74, multiple-comparison test): on average, it took 
them 3.83 + 0.7 sessions to learn the first 1D remappings (Fig. 
3D, blue and green lines). On the other hand, rats that started 
with the diagonal remapping took 11.7 + 2.3 sessions to relearn 
the task, even though they had the help of the visible light (Fig. 
3D, orange). This was a significant difference in learning rate 
(p < 0.0001, multiple-comparison test). These results show that 
the rats clearly used the spatial information available in the dis- 
tributed IR prosthesis. 

We also examined, in more quantitative terms, the pattern of 
errors after left-right remappings. Specifically, we measured 
whether the initial head movement was toward or away from the 
IR source in the first 20 trials of the first remapping session (Fig. 
3F). This measure was also made for the session preceding the 
first remapping, and on the third day of the remapping. As ex- 
pected, the animals turned in the opposite direction directly after 
remapping significantly more often than before (p = 0.0001, 
ANOVA). After learning the remapping, the animals turned di- 
rectly toward the IR source, indicating that they had fully adapted 
(p = 0.0003, ANOVA). 

Interestingly, once a rat experienced its first remapping, it learned 
subsequent remappings significantly faster, suggesting the existence 
of a central metaplasticity mechanism (see Discussion). Figure 3E 
shows the learning curve for all secondary remappings in animals 
after they reached criterion (85% correct) on their first remapping. 
Interestingly, the speed of learning for secondary remappings did 
not vary for the 1D versus 2D diagonal cases (p = 0.79, ANOVA). 
Rats reached the 85% correct threshold in 2.17 + 0.2 sessions. Not 
only were secondary remappings learned faster than initial diagonal 
remapping (p < 0.0001, multiple-comparison test), but they were 
also learned faster than the first 1D remappings (Fig. 3F; p = 0.048, 
multiple-comparison test). 

The previous results address the effect of remapping in an 
already trained animal, but what if we change how they are 
trained initially? That is, is it important that they start with the 
most “natural” mapping in which (for example) the front-right 
detector activated stimulates electrodes in the region of S1 repre- 
senting the front-right whiskers (Fig. 1)? To answer this, we 
trained five rats with the diagonal mapping condition from day 1. 
The overall percentage correct in the first five sessions for rats 
initially trained on the normal mapping was 64 + 7%, while in 
the diagonal-mapped animals it was 40.0 + 10% correct, showing 
a significant difference (Fig. 3H; p = 0.0007, two-sided tf test). 

While the diagonally mapped animals took longer to learn the 
task (Fig. 31), they did ultimately reach a best five-session perfor- 
mance of 98.4 + 0.43% correct, which did not differ significantly 
from the best five-session performance of the 11 normally 
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Figure 3. —Disrupting the topographic IR map temporarily impairs task performance. A, Once animals had learned the task to criterion (at least 85% correct), we performed a series of remapping 
experiments in which we changed the mapping from IR sensor to stimulating electrode. This panel illustrates the four different mapping conditions we used, such as switching hemispheres, 
front-back switches, or both. B, C, Examples from two rats run through multiple remappings. After the rats were proficient with one mapping of the stimulation, anew mapping was introduced. Each 
remapping type is depicted ina different color. The performance on the day before the first remapping is depicted by the green dot at session 1. Interestingly, all animals that started with the diagonal 
remapping stayed at chance performance, and in those animals we added visual cues as an additional aid (this is indicated by the black x). Also, see Movie 2. D, A summary slide of all learning curves 
for the first remapping and all subsequent remappings. Data are collected from nine rats in three groups. Each group started with a different type of remapping (left-right, front-back, or diagonal). 
Subsequent remappings were chosen pseudo-randomly for all rats in all groups. One-dimensional remappings (left-right and front-back) were learned significantly faster (p < 0.0001, multiple- 
comparison test) and without the guidance of visible light. No difference in learning speed was found between the left-right and front-back remappings ( p = 0.74, multiple-comparison test). It took 
3.83 + 0.7 sessions to learn the first one-dimensional remapping. The first diagonal remapping took 11.7 + 2.3 sessions. All rats needed visible light as guidance when confronted with a diagonal 
remapping initially. We started visible light when the rats performed worse than 10% on their fourth day of remapping. E, Analysis of error patterns in the first 20 trials to during left-right remapping. 
For those trials in which IR lights were activated to the left or right of the animal, the plot shows the percentage of trials in which the animal turned toward the (Figure legend continues.) 
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IR prosthesis 
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Movie 2. __Left-right remapping example, related to Figure 3. The connection between the 
sensors and the stimulation channels was switched between left and right (Fig. 3A). During the 
first trials, the rat was expecting the normal mapping and turns in the wrong direction when an 
IR light is activated to the left or right. After training (in this case, ~ 18 min), the rat has begun 
to adapt to the new mapping. (The lights on the left side of the arena are used for off-line image 
processing and are not relevant for the task.) 


mapped rats (p = 0.718, two-sided tf test). Interestingly, the av- 
erage response latencies of the diagonally mapped rats (2.56 + 
0.168 s) never matched those of the rats with the normal topog- 
raphy (1.73 + 0.07 s; p = 0.0011, two-sided t test). 

Note that, while there was a weak but significant correlation in 
learning rates between the learning time of the visual task and the IR 
task (p = 0.0494, Spearman correlation), we found no further cor- 
relations between learning rates in subsequent subexperiments. 


Stimulus population vectors 
In the next series of experiments, we recorded neuronal activity in 
S1 while rats performed the IR discrimination task. As shown in 
Figure 4A, to work around the stimulus artifact that occurs dur- 
ing electrical microstimulation, we modified the task to stimulate 
S1 intermittently. Specifically, after sampling the IR level in a 
sensor, we stimulated for 75 ms at the corresponding frequency, 
followed by a 100 ms quiet period. During this period of quies- 
cence, we were able to record S1 neuronal activity. This was 
done in all four stimulation locations, as depicted in Figure 4B. 
Quantitatively, every 175 ms there was a four-element vector of 
frequencies S = <f,, fi, f;, f,> delivered to four regions of S1 (Fig. 
4B,C). Figure 4B represents five such stimuli, depicted at times 
peer I 

To compactly depict the electrical stimulation delivered to the 
brain in one of these distributed bursts, we represented the full 4D 
representation of the stimulus into a more easily visualizable 2D 
space using a stimulus population vector. This population vector is 
defined exactly like those popularized in the motor control literature 


<_ 


(Figure legend continued.) _ correct light (see Materials and Methods). Performance was sig- 
nificantly worse during the first 20 trials on the first day of remapping compared with the day 
before the first remapping and the third day of remapping (p = 0.0001 and p = 0.0003, 
respectively, ANOVA). F, After learning the first remapping, all subsequent remappings were 
learned in 2.17 + 0.2 sessions, regardless of the type of remapping (p = 0.79, ANOVA). G, 
Number of sessions the rats needed to reach the 85% threshold for different remapping condi- 
tions. After rats learned one remapping, they learned subsequent remappings significantly 
faster (p < 0.0001, multiple-comparison test). This suggests that a metaplasticity mechanism 
isin play. H, Mean + SEM performance on the first five sessions for animals trained with normal 
mapping from the start (NW = 8) and those trained with diagonal mapping (NW = 5 animals). The 
animals trained with diagonal mapping performed significantly worse (p = 0.007, two-sided 
t test). J, Learning curve for normally mapped vs diagonally mapped animals, using the same 
groups of animals as in G. The dotted line marks the 85% threshold. 
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(Georgopoulos et al., 1986). The stimulus population vector is the 
weighted sum of the location of each IR sensor (Fig. 4D), where the 
location of each sensor is weighted by the stimulation frequency in its 
corresponding stimulating electrode. Quantitatively, the stimulus 
population vector P in response to S = <f,, fy fy fa> is as follows: 


P(S) = Dfisi (1) 


where s; is a vector pointing in the direction of IR detector i (e.g., 
the rear-left detector location is represented by <—1, —1>. 

For instance, when the front two IR detectors are strongly 
activated, the population vector is at the top of the graph (Fig. 
4A—D, time point T5). Quantitatively, the rear-left IR sensor 
(sensor 3) is represented by the vector <—1, —1>, and if we 
stimulate at 200 Hz in its corresponding stimulating channel, 
then the contribution from sensor 3 to the population vector will 
be <—200, —200>. The contribution from all four sensors is 
summed to yield the full population vector. Figure 4, B and C, 
works through an artificial example of how to visualize five such 
vectors. This is meant for illustrative purposes to help explain the 
population vector concept. Figure 4E shows, in red, the set of all 
actual population vectors observed in all 20 sessions in three rats. 
The set of all possible population vectors in our stimulus set is 
shown as gray points in Figure 4E. The upper bound in each 
direction is indicated with the black diamond. We stimulated 
between 0 and 425 Hz in an individual channel, and these limits 
determined the upper bounds of 850 Hz for any individual pop- 
ulation vector (Eq. 1). 

Note the relative paucity of such vectors in the posteromedial 
region, which exists because the animal rarely backs up into an IR 
source. That is, the two rear sensors are rarely activated strongly 
together: the animal does not approach the target moving backward, 
but typically orients toward the target and then runs forward. 

To show how the stimulus population vectors change over 
time within a trial, Figure 4F shows the set of population vectors 
that occurred in one session, with contiguous substimuli from a 
trial connected by lines (the red points are error trials, green are 
correct trials). The lighter points/lines represent the stimuli that 
occurred earlier in the trial, and darker rendering represents the 
later stimuli. Note the characteristic shape of the curves: stimuli 
tended to start around the origin (<0, 0>) in the space of pop- 
ulation vectors, and as the animal approached the correct IR 
port, the population vector shifted to the top part of the graph, 
indicating that the front two IR sensors were strongly acti- 
vated. The mean of the first and last three stimuli in the session 
are shown in black. On average, 14 + 0.53 stimuli occurred on 
each trial (Fig. 4G). 


Neuronal peristimulus time histogram during the execution 
of the IR detecting task 
We used multiple measures to quantify the neuronal responses in 
S1 during a session. Figure 4H shows one measure: the mean 
neuronal population peristimulus time histogram (PSTH) in re- 
sponse to the first three and last three stimuli in a session. The 
times of actual stimulation are numbered 1—6 in the PSTH, 
where the first 3 are aligned to the onset of the cortical stimula- 
tion and the last 3 to the offset. During the 75 ms stimulation 
periods, the PSTH is zero because of stimulus artifact, and then 
there is a 100 ms period of neuronal activity recording after the 
stimulus ends (Fig. 4A). 

The set of all such neuronal population PSTHs obtained in a 
given recording session is shown in Figure 5, where the location 


187 


Hartmann, Thomson et al. e Embedding an Infrared Representation in S1 J. Neurosci., February 24, 2016 + 36(8):2406 —2424 * 2415 


A 


Stim Only =~ Stims + Record B 
Channel 1 
=z = 
= x 
2 = = Channel 2 
S e _ 
pa J) 
= = i 
ro) rw 2 Channel 3 
=) 
w) 
Time [s] Timetsl Channel 4 


Time [s] 


Cc D E  g50 


200 
Population Sensor 2 Sensor 1 — 510 
4-Vector —> Vector ca = 
Tl <0,0,50,50> <0,-100> 5 2 
3 5 170 
T2  <0,0,50,0> < -50,-50 > g 7 g 0 
T3 <10,50,10,0> <-50,50> § ee 
g 2° 
T4 <50,50,0,0> <0, 100 > -100 = 
TS <100,100,0,0> <0,200> Sensor 3 Bensor 4 —510 
-200 
-200 “100 0 100 200 
Mediolateral 850 1 


—850 —510 -170 0 170 510 850 
Mediolateral[Hz] 


F850 G 
510 
z x 0 5 10 15 20 25 30 
= 170 , K Number Of Stimuli 
S H 
© 0 ’ 35 
3 3 
= 170 i 
g / #25 
() =] 
< a 
~*~ 15 
-510} ’ , ‘3. 
1 
| \ “4 H(S) = 6 bits a 
a 0 
—850 —510 -170 0 170 510 850 -05 -0.25 0 0.25 05 0.75 1 1.25 15 
Mediolateral [Hz] Time [s] 


Figure 4. Quantifying the stimulus projected into $1. A, Illustration of change in protocol used when simultaneously stimulating and recording. When just stimulating, we updated the 
stimulation frequency in each channel every 50 ms based on the IR levels in its corresponding detector (Fig. 1F). In animals in which we stimulated and recorded, we sampled the IR levels 
approximately every 175 ms, and delivered stimulation for 75 ms in each stimulating channel, and then turned off the stimulators for 100 ms to facilitate recording. B, Each stimulation pattern can 
be represented by a sequence of four-element vectors $ of microstimulation frequencies in four locations in $1. The panel shows five simplified patterns of microstimulation. C, Conversion from 4D 
vector representation of stimulus to the 2D population vector, where each time point (T1—T5) corresponds to one of the five time points in B. The actual numbers (0-100) are arbitrary and are used 
only to clarify the procedure. D, The same four population vectors from C are depicted as locations in the 2D population vector space. Again, this is for illustration only, to facilitate understanding of 
simple stimulus population vectors, and does not include real data. All four sensor location vectors are indicated for reference, as are five population vectors (labeled T, . . . T;). The black square is 
the outline of the bound of all possible stimulus population vectors, which are discussed more in D. E, The set of all possible population vectors (gray points), with the set of all actual population 
vectors over all recording sessions overlaid (red points; 19 sessions in three animals, with 171 + 53 trials per session and 14 = 0.53 stimuli per trial). Note that the actual vectors (red) tend to cluster 
inthe medial anterior portion of the space, with relatively few population vectors on the extreme lateral and caudal regions of the space of population vectors. The range from 0 to 850 Hz results from 
the maximum stimulation frequency of 425 Hz per channel, which can sum up to 850 Hz in the two-dimensional transformation. F, The set of actual population vectors (Figure legend continues.) 
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S1PSTHin response to stimulation sequences. The mean response to the first three, and last three, stimulus vectors, as depicted in Figure 4F. The arrows from the brain show the location 


of the recording electrodes. Each PSTH follows the conventions shown in Figure 4H. The inset shows the spike count z-score as a function of the number of stimulating channels activated. This is a 
typical profile, with maximum response occurring when two channels are coactivated. The z-score was calculated by comparing the spike count after stimulation, and we used the mean/SD of spike 
counts before trial onset (calculated over all trials; data are from N = 3 animals, 20 recording sessions, 299 multiunit recordings). Even when no stimulating channels are active, the response is 


typically higher than in the baseline period before the trial begins. 


of each recording microelectrode is also indicated. We consis- 
tently found the IR-evoked neuronal responses to be broadly 
distributed across the recording microelectrodes in all four quad- 
rants of S1. The inset on the right side of Figure 5 depicts the 
mean neuronal response as a function of the total number of 
stimulating channels activated. This graph shows that the larger 
numbers of spikes were produced by S1 neurons until two IR 
channels were activated. Beyond that point, when three or four IR 


= 


(Figure legend continued.) delivered in a single session, with those stimuli in the same trial 
connected by lines. Red indicates error trials, while green shows correct trials. Lighter circles 
represent earlier stimuli in the trial, and more saturated colors indicate later stimuli. The black 
spots represent the mean value of the first three and last three stimuli in the session. The early 
stimuli hovered around the origin, while the final three stimuli tended to be closer to the rostral 
pole, indicating the two front IR channels (and stimulating electrodes) were highly activated. In 
this session. There were 6.0 bits of entropy contained in the stimulus set. The light gray circles 
indicate the set of all possible population vectors as before, and are included for reference. G, 
Count histogram of the number of stimuli shown on each trial, overall sessions. The distribution 
is approximately bimodal, with the peak centered at ~25 indicating those trials in which the 
animal let the trial time out without poking, and reached the upper limit of net stimulation. H, 
PSTH in response to the first three and last three stimuli in the same session, as indicated in D. 
The stimulation periods are indicated by the numbers 1— 6, and time is relative to the onset of 
the first stimulus. Note that this PSTH does not represent the response to the exact same 
stimulus each time: while the first three stimuli tend to cluster around the origin and the last 
three tend to cluster in the mediorostral region, but there is a good deal of variability (D). This is 
simply a way to get a quick overview of the types of responses we see. We indicate the potential 
break in time between the first three and last three stimuli on the x-axis with the slashes. 


channels were activated, $1 neuronal firing tended to decrease, 
though this change was not significant (ANOVA, p > 0.05). 


Infrared neuronal receptive fields in the rat S1 cortex 

While the PSTH in response to the first and last three stimuli gives a 
convenient visual representation of neuronal activity during the 
task, it is also a relatively incomplete sample of the full range of 
neuronal firing patterns. To give a full representation of how S1 
neurons responded to IR stimuli, we introduced a new measure- 
ment: the IR-RF. 

The IR-RF is the mean S1 neuronal firing response magnitude 
(spike count), as a function of the stimulus population vector, for 
all of the IR stimuli shown in a session. Figure 6A shows all of the 
IR-RFs for the same S1 neuronal units depicted in Figure 5. The 
mean value for each stimulus is color coded in a contour plot 
defined over the set of stimuli, where red lines depict peak re- 
sponse values. For instance, unit 019i has a peak at the most 
anterior/medial portion of the stimulus space, indicating that it 
responds maximally when the two front IR sensors are strongly 
activated (Fig. 4), while unit 0317 has its IR-RF center located in 
the posterior-right region of the space. 

The contour plots suggest that the locations of the IR-RF 
peaks are quite variable from neuron to neuron. However, they 
are not randomly distributed. For the 179 S1 units recorded un- 
der the normal mapping, the distribution of IR-RFs peaks tend to 
cluster near where the IR stimuli clustered, in the anterior/medial 
space of the IR stimulus population vectors (Fig. 6B, red ellipse, 
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Figure 6. _ IR receptive fields. A, Mean number of spikes in response to different stimuli. The stimuli are represented as population vectors 
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Fig. 6), and the mean spike count in the 60 ms time 


window is represented by isocontour lines (low rates in blue, high in red: see color bar). Same session and units as used in Figure 5. The small red point on each contour plot shows the point where 
the receptive field is maximized. B, Distribution of locations of RF peaks in all units with the normal mapping (179, multiunits total). They tend to be located along (Figure legend continues.) 
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shows the 95% confidence ellipse for the mean of the distribution 
of IR-RF centers). For comparison, Figure 6C plots the distribu- 
tion of all stimulus population vectors presented to all animals 
that were recorded. Quantitatively, the 2D correlation between 
the IR-RF centers and the actual distribution of IR stimuli was 
0.44, a significant correlation (p < 0.001, F test). Altogether, 
these data suggest that the overall distribution of neuronal IR- 
RFs tended to reproduce the spatial distribution of stimulus pop- 
ulation vectors used, though with a slight preference for larger 
stimuli (i.e., closer to the borders of the diamond) on the medio- 
lateral axis. 

To test the hypothesis that response preferences in S1 neurons 
tend to match those of the stimuli delivered, we then compared 
the IR-RFs for three animals in which the mapping between front 
and rear channels was reversed (104 units). If neurons tend to 
naturally prefer activation of the anterior channels (Fig. 1B), then 
we would expect the IR-RF centers to be localized in the posterior 
portion of the population vector space when the front and back 
channels are reversed. However, this is not what we observed. 
Instead, the IR-RFs continued to favor the front stimuli, with the 
mean IR-RF center located at <— 126.3, 205.4>, anda significant 
2D correlation between IR-RF centers and stimulus distribution 
(2D correlation coefficient, 0.20; p = 0.003, F test). This confirms 
the hypothesis that response profiles, as summarized in the IR- 
RFs, tend to match the distribution of stimuli actually delivered 
during the task. 

There was not a simple relationship between stimulating elec- 
trode location (or even hemisphere) and RF center. Neurons did 
not simply respond maximally to localized stimulation. Neurons 
tended to respond most strongly to stimulation delivered in two 
locations (Fig. 5, inset), often between hemispheres. Of 179 S1 
units recorded with the normal mapping, 65% (117 of 179 S1 
units) had receptive field centers located in a region in which at 
least two microstimulators were active (Fig. 6 D, E). Of those with 
a preference for stimulation at two or more locations, 58% (68 
of 117 S1 units) had a preference for stimulation across hemi- 
spheres. 

We next examined the effects of learning on RF magnitude 
and diameter. Figure 6F displays the IR-RF peak magnitude when 
the behavioral performance of the animals was fairly poor 
(<40% correct) versus when rats reached excellent behavioral 
performance (>90% correct). This did not change with learning 
(p = 0.91, tf test), suggesting that $1 responds with robust re- 
sponses very early in training, which is what we observed infor- 
mally even on the first day that animals were training on the IR 
task. Figure 6G depicts the IR-RF diameter as a function of be- 
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(Figure legend continued.) the midline, and rostral. Thatis, they tend to favor the activation of 
the two front IR channels. The histograms on the sides show the marginal frequencies. The red 
spot shows the mean, and the circle around it shows the 95% Cl for the mean (see Materials and 
Methods). The mean RF centers are also indicated for the four different recording locations. C, 
Distribution of locations of actual stimulus vectors presented in all experiments (see Fig. 4£). 
They tend to be located along the midline, and rostral, just like the receptive field centers. The 
histograms on the sides show the marginal frequencies. D, Bar plot showing the relative num- 
ber of neurons with preference for stimulation from pairs of stimulating electrodes vs individual 
stimulating electrodes. E, Proportion with preference for pairs is significantly greater than 
chance (p = 0.003, x” test). Angular preference of all neurons are from B. Angular histogram 
shows the relative number of neurons with IR-RF centers at different angular locations. F, Peak 
response magnitude (z-score, with mean/SD spike count calculated from a 60 ms baseline 
period over all trials in a given session) does not depend on training (p = 0.49, two-sided t 
test). G, RF size changes with learning. Mean + SEM RF diameter for early- and late-training 
animals are significantly different (p = 0.009, two-sided f test). 
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havioral performance. The IR-RF diameter is a measure of the 
area enclosed by the contour line at 75% of the maximum value of 
the IR-RF (see Experimental procedures). This analysis revealed 
that the IR-RF width actually increased with learning (p = 0.009, 
t test). 


Population discrimination of 75 ms stimulus pulses 

Next, we addressed the question of how well S1 neuronal firing 
patterns could predict the stimulus presented to the animal in 
a given 75 ms epoch. To do this, we applied naive Bayesian 
classifiers. Because many stimulus vectors do not occur often 
enough to build a reliable classifier, we narrowed our classifi- 
cation problem to the eight most frequent IR stimulation pat- 
terns in each session (a 3 bit classification task). We then ran 
the classifier on different numbers of S1 neurons, from 1 to 14 
units at a time, depending on how many we were able to record 
simultaneously. Figure 7A displays the mean performance of 
the classifier as a function of the number of neurons used to 
train and test the classifier: a neuron-dropping curve (Wess- 
berg et al., 2000; Lebedev, 2014). The red line depicts chance 
level. As one can see, the percentage of trials correctly classi- 
fied by the Bayesian algorithm rose from 20% (using one unit) 
up to 52% when the spike counts from 14 units were combined 
into a single vector. 

When we broke our analysis to look into high- and low- 
performing sessions, as described in Figure 6, we observed that 
neural ensemble discrimination of single-trial IR stimuli was 
significantly better in highly trained animals (Fig. 7B; p = 
0.048, two-sample Kolmogorov—Smirnov test). This suggests 
an improvement in neuronal discrimination power as a func- 
tion of learning. 


S1 responses to uninformative microstimulation patterns 
Electrical stimulation of the brain, outside of any task context, 
clearly produces significant responses, and can even induce 
drastic plasticity (Nudo et al., 1990; Recanzone et al., 1992; 
Maldonado and Gerstein, 1996). To examine whether the re- 
sponses we observed depended on the microstimulation pat- 
terns providing information to be used to perform the IR 
discrimination task, we delivered identical microstimulation 
patterns in two control animals, but in a way that was unre- 
lated to any task (see Experimental methods). Briefly, we 
pulled stimulus sequences from rats that actually performed 
the task, and delivered them at random times to animals that 
performed a normal visual version of the task (no IR light), 
such that microstimulation provided no information about 
the stimulus in each port. This allowed us to reproduce the 
general context of stimulation, but remove its task relevance. 

In both animals, after 2 weeks of training with task- 
irrelevant microstimulation, S1 sensory-evoked responses 
were drastically suppressed. Figure 8A shows an example from 
one animal in which we recorded 14 S1 units simultaneously. 
The mean response to the first three and last three stimuli is 
shown in Figure 8B, showing the gross suppression effect (Fig. 
8B, top) compared with the normal animals (Fig. 8B, bottom). 
Overall, the mean + SEM of the maximum response to all 
stimuli was 1.4 + 0.13 when the stimulation was uninforma- 
tive, and 17.4 + 1.6 when informative, showing a significant 
difference (p < 0.001; Fig. 8C). Similarly, the mean response 
magnitude as a function of the number of stimulating chan- 
nels activated was effectively a mirror image of the case with 
informative stimulation, with values reflected across the x-axis 
(Fig. 8D). 
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Figure7. — Discrimination of microstimulation patterns in S1.A, We selected eight of the most 


frequent population vectors for each session and used a Naive Bayes’ classifier to predict the 
stimulus based on neuronal activity in $1. Plot shows the percentage of population vectors 
classified correctly as a function of the number of units used to predict the stimulus (mean + 
SEM, V = 20 sessions, 299 multiunits total). The red line is chance (12.5%). Note that this 
represents a differential response to a single 75 ms stimulation bout (e.g., stimulus T3 in Fig. 
4B). On average, there are 11.3 + 5.1 stimuli/trial (mean + SD). B, Same data asin A, but with 
discrimination separated by behavioral performance, as in Figure 6. 


We also examined the ability of the neuronal responses to 
discriminate microstimulation patterns, expecting it to be signif- 
icantly diminished compared with the case when such patterns 
were informative about events in the environment of the ani- 
mal. Surprisingly, these inhibitory responses carried almost as 
much information about microstimulation patterns (Fig. 8E; 
see Discussion). 


S1 neuronal firing responses to whisker deflections in rats 
implanted with IR neuroprosthesis 

We observed strong rapid S1 neuronal sensory-evoked responses 
to whisker deflections in animals trained on the IR discrimina- 
tion task. Figure 9A shows the carbon monoxide-stained section 
through both hemispheres for one animal that was highly trained 
in the task (performance consistently above 95% correct) to show 
the location of the recording electrodes. Figure 9B displays the 
responses of 15 S1 units (multiunit activity), in the same animal, 
to deflections of the right whiskers using an air puff. We observed 
significant responses in all S1 units in the three rats (N = 36 units; 
Griffin et al., 2011). The S1 neuronal response latency was 10.8 + 
0.38 ms, and the mean response duration was 92 + 12.5 ms. The 
response magnitudes were quite large: the mean number of spikes 
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above baseline was 728 + 91 total spikes for multiunit activity 
and 77 + 21 total spikes from 15 single units in the same animals. 
The mean response to all whisker deflections, over all 36 multi- 
unit recordings, is shown in Figure 9C; clear onset and offset 
responses can be observed. These results suggest that $1 has not 
been hijacked by the IR neuroprosthetic (Griffin et al., 2011). 
Instead, S1 seemed to have incorporated an ability to represent 
both stimuli simultaneously into a multimodal map of the tactile 
and IR environment. 


Discussion 

The adult mammalian cortex is able to overcome the biologi- 
cal constraints imposed by the native peripheral sensory trans- 
ducers of an organism, absorbing a spatially distributed 
representation of a completely novel sensory modality. Adult 
rats quickly adapted their behavior to use a panoramic IR 
neuroprosthesis, attaining near perfect performance in a sen- 
sory discrimination task involving the new sensory modality 
(Fig. 2). The cortical neuroprosthesis used in this study is, as 
far as we can tell, the first device to project a panoramic topo- 
graphically distributed projection of the subject’s surround to 
both hemispheres of a primary cortical area. Based on the 
ablation experiments (Fig. 2H), it is clear that rats used all four 
information channels, though none are indispensable. In- 
stead, performance dropped gradually with each IR sensor 
inactivated, but even when only one sensor was active, perfor- 
mance was still significantly above chance (Fig. 2H). 

Note that the use of IR sources was largely arbitrary. We could 
have used any information source that bypassed the natural 
transduction pathways of the rats, as long as a portable sensor was 
available (e.g., magnetic fields; Nagel et al., 2005; Norimoto and 
Ikegaya, 2015). 

In the 1960s, Bach-y-Rita et al. (1969) pioneered sensory sub- 
stitution experiments in which continuous streams of visual in- 
formation were delivered mechanically to the skin on the back of 
blind human subjects, using actuators laid out to preserve the 
topographic organization of the original visual stimulus. While 
our rats needed just a few days to show excellent performance 
with the prosthesis (Fig. 2), in humans there is a great deal of 
variance in the learning curves for peripheral sensory substitu- 
tion devices. The time taken to get used to different peripheral 
devices ranges from 30 h to months (Nagel et al., 2005; Amedi et 
al., 2007; Kim and Zatorre, 2008; Karcher, 2012; Kaspar et al., 
2014). Even compared with the studies with the shortest learning 
times (30—40 h), our rats performed as well or better after 3 h 
(Amedi et al., 2007; Kim and Zatorre, 2008). Hence, we hypoth- 
esize that people who one day take advantage of sensory neuro- 
prosthetic systems that deliver information directly to the CNS— 
particularly primary sensory areas—will learn to use the new 
information faster than those using feedback delivered to the 
periphery. Current motor neuroprostheses typically record a mo- 
tor control signal in the brain and send this to a prosthetic limb 
(Chapin et al., 1999; Carmena et al., 2003; Nicolelis, 2011; Ifft et 
al., 2013). Using the technology described here, the brain could 
be placed in a continuous two-way communication with infor- 
mation from sensors in prosthetic limbs (O’Doherty et al., 2012). 
Our data suggest that when designing prosthetic systems for pri- 
mary sensory areas, it would be prudent to exploit preexisting 
topographic maps, as rats learned faster when stimulating chan- 
nels were aligned with the somatotopic map of the barrel field 
(Fig. 31). One explanation of this might be that the brain already 
contains the downstream connectivity pattern required to extract 
spatial information from the topographic map in S1, and this 
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Figure 8. Uninformative stimuli produce inhibitory responses. A, PSTH in all neurons recorded in an animal receiving noninformative microstimulation sequences. Specifically, we 
delivered microstimulation sequences pulled from animals performing the actual IR discrimination task, but delivered at random times, to an animal doing an unrelated visual task. B, 
Mean response magnitude to the first three and last three stimuli in two animals, six sessions, 78 units. Red bars, Response when ICMS is not task relevant; black bars, mean response 
when ICMS is used for IR discrimination. Note the inhibitory response to uninformative ICMS. C, Mean peak response magnitude in all neurons in which ICMS was not task relevant vs when 
it was task relevant. D, Mean + SEM response magnitude (z-score) as a function of number of units active for the task-irrelevant (red) and task-relevant (black) ICMS. Again, note the 
difference in magnitude and the sign of the response. E, Surprisingly, the inhibitory responses carried just as much information about stimulus identity. The discrimination performance 
as a function of the number of units is shown for animals stimulated during the IR task (black), outside the task context (red), and with stimulus labels shuffled (gray). This suggests that 
information about microstimulation patterns is present in latent form, ready to be exploited by the brain when it becomes task relevant. 
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S1 neurons that respond to ICMS are still responsive to whisker deflections. A, Flattened cortical sections through both $1 hemispheres in one animal show the location of electrodes. 


The asterisks mark the electrode locations. B, Multiunit response to whisker deflections is robust for all units. Figure shows PSTH in units from the same rat depicted in A, after training on the IR 
discrimination task, upon deflection of the right whiskers with a 200 ms air puff delivered as the rat sat in the behavioral chamber. C, Mean multiunit response to whisker deflection, in all 36 units 


recorded in three animals. Bin width, 1 ms. 


makes the natural mapping easier to acquire (Diamond et al., 
1999; Harris et al., 1999). 

Previous work has shown that the barrel cortex integrates in- 
formation across hemispheres to perform simple spatial discrim- 


ination tasks (Krupa et al., 2001; Shuler et al., 2001). Our results 
show that interhemispheric stimulation patterns can be readily 
discriminated using a topographically distributed multisite 
ICMS in a very short time. 
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Figure 10. Voltage traces from four IR sensors. Representative mean + SEM sensor output 
as a function of time during trial. Time 0 is the time at which the rat selected a port. Note that 
~1.5 s before the end of the trial, the two front sensors rapidly saturate as the animal runs 
toward the port, while the signals from the rear sensors drop. However, when the animal gets 
very close to the port, the IR source is broad enough to actually stimulate the rear IR sensors, as 
you can see with the final peak close the time of the poke. See Discussion. 


Neuronal coding of IR stimuli 

After training with the IR neuroprosthesis, S1 neurons simulta- 
neously encoded both tactile and IR stimuli in a synthetic multi- 
modal map (Fig. 9). Future experiments will be needed to 
determine the effects of this new IR modality on the ability of the 
animals to perform tactile discrimination tasks using their whis- 
kers (Ni and Maunsell, 2010). 

One surprising finding was that inactivation of rear-facing chan- 
nels had a more deleterious effect on IR discrimination than inacti- 
vating the forward-facing IR sensors. The front channels were the 
dominant source of inputs to the system for the duration of most 
trials (Fig. 4F), so why did animals seem to preferentially use the rear 
channels? One possibility is that the rats use heightened activity in 
the back channels at the end of a trial as an indicator that they are 
getting very close to the correct port. Indeed, Figure 10 shows that 
the mean output of the rear voltage sensors actually increases when 
they are very close to the correct port. 


Neurobehavioral plasticity 

There were multiple types of plasticity exhibited here. Clearly, 
animals had to learn to use the new information source projected 
to S1. This demonstrates that, throughout the life of the animal, 
the brain continues to monitor the statistical structure of its in- 
puts. Even at a behavioral level, animals made subtle adjustments 
to their behavior to more closely align the sensors on their head 
with the IR sources in their environment (Fig. 2 E,F). It will be 
crucial, in a future study, to track behavior at finer spatiotempo- 
ral scales, and fully quantify the relationships among stimulation, 
behavior, and the neuronal response in S1. 

At a finer-grained level, our data suggest that responses in the 
adult cortex continuously adapt so that they will match the struc- 
ture of those inputs. This was suggested by our original recording 
data, in which RF centers matched the general distribution of 
stimuli (Fig. 6). This was confirmed in an independent set of 
experiments in which IR-RF centers continued to match the in- 
put distribution, even when the front and rear channels were 
swapped. These results are similar to what is observed in devel- 
opment (Sengpiel et al., 1999). 
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Future experiments should probe just how far this plasticity 
can be pushed. For instance, it would be helpful to deliver pre- 
defined test pulses in awake and anesthetized animals to deter- 
mine the degree to which responses to those test pulses indeed 
change over time. 

While the brains of rats were quite capable of adopting new 
behavioral strategies to harvest information from their IR envi- 
ronment, the remapping experiments examined the ability of 
animals to change that strategy when pushed into unusual re- 
gimes it would not normally experience. Thereby, we follow ex- 
periments like those using the retinal inversion lenses by Stratton 
(1896), the rewiring studies of Sur et al. (1988), frogs with rotated 
eyes by Udin (1983), and the work on the effects of adding a third 
eye to the frog by Katz and Constantine-Paton (1988). We are not 
aware of such inversion/remapping experiments being per- 
formed in previous sensory substitution systems in humans. 
While rats initially performed poorly after a remapping was ap- 
plied (Fig. 3A), they quickly adapted when the remapping was 
relatively simple (front-back or left-right switches). However, 
when the topography was more complex (the diagonal remap- 
ping), they needed the help of visual cues to relearn the new 
mapping (Fig. 3B, D). This illustrates the sensitivity of the brain 
to pre-existing topographic maps, even artificial maps, suggest- 
ing that that it cannot learn arbitrary input/output mappings. 

Interestingly, once animals learned one remapping, they 
learned subsequent remappings significantly faster (Fig. 3G). 
This suggests that the remapping experiments are triggering a 
central metaplasticity mechanism, such that animals literally 
learned to learn the new mappings. Such a boost in learning rate 
after being exposed to demanding or enriched environments and 
tasks was suggested by Hebb (1949). While metaplasticity may 
involve the modulation of molecular-level synaptic plasticity 
mechanisms (Abraham and Bear, 1996; Rebola et al., 2011), there 
are likely also higher-level cognitive mechanisms at work, such as 
changes in the expectation of the animal about the task. With the 
present rodent model, we can more thoroughly explore these 
options, for instance by blocking NMDA receptors during re- 
mapping experiments. 

One surprising effect was the influence of the task relevance of 
microstimulation on responses in S1 (Fig. 8). We delivered un- 
informative microstimulation patterns in rats performing an un- 
related visual task. We expected similar response properties in 
both groups of animals, as it seems that passive stimulation is 
sufficient to induce significant response plasticity (Nudo et al., 
1990; Recanzone et al., 1992; Maldonado and Gerstein, 1996). 
Instead, we saw drastically suppressed responses in the majority 
of units examined (Fig. 8). This complements previous observa- 
tions that responses in primary sensory areas are strongly depen- 
dent on task context (Fanselow and Nicolelis, 1999; Fanselow et 
al., 2001; Nicolelis and Fanselow, 2002; Niell and Stryker, 2010; 
Fuetal., 2014). Our results suggest that responses depend not just 
on the statistical structure of the inputs, but also on their rele- 
vance. The causes of the suppressed activity could be a combina- 
tion of attentional effects, differences in movement patterns 
when the information is not relevant, or other top-down cortico- 
cortical effects on S1 processing (Pais-Vieira et al., 2013). 

Curiously, these suppressed responses also carried significant 
information about the stimulation patterns delivered to the brain 
(Fig. 8E). Hence, while the S1 neuronal responses to uninforma- 
tive stimuli were suppressed, the information remained latent in 
patterns of inhibitory activity. We hypothesize that the cortical 
response profile would become excitatory when the patterns of 
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stimulation become informative about events significant to the 
animal. 

Overall, this study highlights the remarkable plasticity of the 
adult mammalian brain. The brain seems to remain exquisitely 
tuned to different information sources well past any classic criti- 
cal period in development. With the addition of more informa- 
tion channels, the ability of adult rats to exploit the information 
only improved (Fig. 2). Further, $1 neurons showed clear prefer- 
ences for those stimuli statistically most likely to be delivered 
during the task, suggesting that the cortex continuously tunes its 
responses to the statistical structure of its inputs. However, the 
brain did not indiscriminately respond to the most likely stimuli. 
Rather, S1 responses were actually suppressed when statistically 
probable stimuli were not informative about task-relevant fea- 
tures in the environment of the animal. That is, in addition to 
activity-dependent plasticity mechanisms, powerful relevance- 
sensitive mechanisms strongly sculpt cortical activity at a given 
time. 
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Abstract 


Adult rats equipped with a sensory prosthesis, which transduced infrared (IR) signals into electrical signals 
delivered to somatosensory cortex (S1), took approximately 4 d to learn a four-choice IR discrimination task. Here, 
we show that when such IR signals are projected to the primary visual cortex (V1), rats that are pretrained in a 
visual-discrimination task typically learn the same IR discrimination task on their first day of training. However, 
without prior training on a visual discrimination task, the learning rates for S1- and V1-implanted animals 
converged, suggesting there is no intrinsic difference in learning rate between the two areas. We also discovered 
that animals were able to integrate IR information into the ongoing visual processing stream in V1, performing a 
visual-IR integration task in which they had to combine IR and visual information. Furthermore, when the IR 
prosthesis was implanted in S1, rats showed no impairment in their ability to use their whiskers to perform a tactile 
discrimination task. Instead, in some rats, this ability was actually enhanced. Cumulatively, these findings suggest 
that cortical sensory neuroprostheses can rapidly augment the representational scope of primary sensory areas, 
integrating novel sources of information into ongoing processing while incurring minimal loss of native function. 


Key words: Behavior; multisensory integration; neuroprosthesis; plasticity 


Significance Statement 


Using a sensory neuroprosthesis that projects information from the IR environment to primary sensory 
areas, we show that adult rats can rapidly integrate completely novel sensory information into preexisting 
cortical maps. When the prosthesis is implanted in V1, animals can learn to perform a multimodal integration 


task, fusing IR and visual information that is simultaneously superimposed on the same cortical area. When 
the prosthesis is implanted in S1, the tactile function of S1 is left undisturbed, and often enhanced. Hence, 
it is possible to merge multiple streams of information onto the same primary cortical area without 
compromising its original function. This is auspicious for the development of sensory prosthetic systems for 
adult victims of brain injury. 
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Introduction 

A fundamental goal in neuroscience is to delineate the 
mechanisms and limits of adult neurobehavioral plasticity. 
This task is important for both basic neuroscience and 
modern rehabilitative medicine (Lledo et al. 2006). This 
question was previously addressed using a real-time 
closed-loop sensory prosthetic system that allowed re- 
searchers to investigate how adult mammals responded 
when information from a completely new source, infrared 
light (IR), was projected to primary sensory regions 
(Thomson et al. 2013; Hartmann et al. 2016). In this cor- 
tical neuroprosthesis, the output of four head-mounted IR 
detectors was coupled to topographically distributed 
stimulating electrodes chronically implanted in the so- 
matosensory cortex (S1) of adult rats (Thomson et al. 
2013; Hartmann et al. 2016). As with ablation-induced 
sensory-cortex rewiring in newborns (Frost and Metin, 
1985; Sur et al. 1988), adult rats readily learned to use this 
new source of information, ultimately performing as well 
as in corresponding visual discrimination tasks. Similar 
results were also observed when information from the 
geomagnetic environment was projected to V1 (Norimoto 
and Ikegaya, 2015). 

Such initial results with cortical prosthetic systems 
raised several key questions that we presently address. 
One, because of the facility with which S1 absorbed 
information about IR sources, we postulated that different 
primary sensory areas should display similar levels of 
sensory plasticity. We advance this thesis based on pre- 
vious results, but also on anatomic grounds: similar ca- 
nonical microcircuits are iterated across the neocortical 
mantle (Haeusler and Maass, 2007; Kouh and Poggio, 
2008; Habenschuss et al. 2013; Harris and Shepherd, 
2015). To test this equipotentiality of plasticity hypothesis, 
we implanted stimulating electrodes in primary visual cor- 
tex (V1) of adult rats to measure their ability to use the IR 
neuroprosthesis and compare this with the performance 
of animals that used an S1-based prosthesis. 

In the original rewiring experiments, performed in new- 
borns, the transformed areas received inputs from only 
one sensory modality-for instance, rewired A1 received 
only visual inputs (Sur et al. 1988; von Melchner et al. 
2000). In contrast, in animals using the IR prosthesis, both 
native and IR information was simultaneously available to 
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the primary sensory area implanted with the prosthesis, 
and this allowed us to ask additional questions about how 
the brain handles the simultaneous superposition of mul- 
tiple sources of information. 

For instance, can the brain integrate both information 
sources simultaneously to perform a multisensory integra- 
tion task? (Deneve and Pouget, 2004; Ghazanfar and 
Schroeder, 2006; Macaluso, 2006; Stein and Stanford, 
2008; Klemen and Chambers, 2012). To examine this 
question, we developed a visual-IR multisensory integra- 
tion task, in which animals were presented with multiple 
visible and IR cues simultaneously, and rats were rewarded 
only if they ignored distractor visual and IR cues, and se- 
lected the stimulus that combined both visible and IR lights. 
As far as we can tell, no previous sensory prosthetic system 
has been tested with a behavioral task with such a high 
degree of difficulty, requiring subjects to integrate informa- 
tion from two qualitatively different sources simultaneously 
superimposed onto the same cortical area. 

Further, when a new source of information is projected 
to a cortical area, does this have unintended deleterious 
side effects? There are multiple lines of evidence that 
intracortical microstimulation (ICMS), by itself, is a pow- 
erful inducer of cortical plasticity (Recanzone et al. 1992; 
Maldonado and Gerstein, 1996; Godde et al. 2002). How- 
ever, there is also evidence from monkeys that repeated 
microstimulation in V1 can impair performance on visual 
detection tasks (Ni and Maunsell, 2010). Also, there is 
evidence from the motor system of monkeys that ICMS 
can even highjack the cortex, displacing its original func- 
tion (Griffin et al. 2011). To directly address the question 
of how a prosthetic system used in a discrimination task 
affects the original function of the implanted area, we 
examined how an S1-implanted prosthesis influenced 
performance on a whisker-based tactile discrimination 
task. The results will have potential clinical implications for 
neuroprosthetic design, as well as more basic implica- 
tions about the limits of plasticity of a given region of 
primary sensory cortex. 


Materials and Methods 


General 

All experiments were performed on adult female Long- 
Evans rats (Harlan Sprague Dawley Laboratories), 250- 
300 g. All animals were treated humanely to minimize 
stress, and the Duke University Institutional Animal Care & 
Use Committee (IACUC) approved all surgical and behav- 
ioral methods. 


Behavioral methods 
Behavior chamber 

The majority of behavioral experiments were performed 
in a cylindrical behavioral chamber (50.8 cm or 20 inches; 
Fig. 1A). A button was placed in the center of the chamber 
that allowed the rats to initiate a trial. In some sessions, 
the chamber included a mechanical button; because the 
mechanical button was difficult to learn to use, it was later 
replaced with an infrared photobeam recessed into the 
floor. Each chamber contained four ports around its cir- 
cumference, initially evenly spaced 90° apart (Fig. 1A). On 
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Figure 1. Methods. A, Setup of the IR behavior chamber. Four reward ports line the walls of the circular chamber. The angle @ 
indicates how far apart adjacent ports are. Inset shows design of each individual port. B, Illustration of the mapping from four IR 
detectors (1-4) to four microstimulating electrode locations (N,-N,) in V1. C, Dorsal view of rat skull showing the placement of the 
S1 and V1 stimulating electrode arrays. The green and blue zones correspond to the S1 barrel field (S1BF) and V1, respectively, and 
these areas were reconstructed from (Paxinos and Watson, 2007). The dark horizontal and vertical reference lines intersect at bregma, 
and each small square in the grid is 1 mm?. D, Transforms from IR levels detected in individual IR detectors to microstimulation 
frequency in individual stimulating channels. The black line is the original transform used in previous articles, and some animals in the 
present article. The red line shows the optimized transform (see Methods). E, Design of electrode arrays consisting of two bundles 
of eight stimulating electrodes placed 2 mm apart (see Methods for more details). F, Flattened cortical slice from animal implanted 
in V1, stained with cytochrome-oxidase. Inset shows detail of macrovibrissae barrels. Oval at bottom (caudal) shows the location of 


tissue damage from the V1 implant. Reference line: 1 mm. 


the inner surface of each port there was a visible LED and 
an infrared LED (Opto Semiconductors; 940-nm peak 
emittance with a range of non-zero emission between 825 
and 1000 nm). The IR sources had an angular width at 
half-maximum of 120°. Each port contained a water spout 
inside a conical recess that also included an infrared 
photobeam, allowing us to detect when the rat selected 
that spout (Fig. 1A, inset). 


Visual discrimination task 

In the visual discrimination task, the animal initiated 
each trial by pressing a button in the center of the behav- 
ioral chamber, and then a single visible light was turned 
on in a single port. The animal received a small amount of 
water if it selected that port. Otherwise, an error tone 
sounded and there was a timeout delay (4-10 s) before 
the next trial could be initiated. We considered threshold 
performance to be when an animal reached 85% correct 
and performed at least 150 trials in a session, at which 
point the animal was ready to be implanted with stimulat- 
ing electrodes and learn the IR discrimination task. 

A total of 35 rats were trained on the visual discrimina- 
tion task (21 to compare S1 and V1 learning rates on the 
basic IR discrimination task, 6 to examine visual-IR inte- 
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gration—3 in S1 and 3 in Vi—and 8 to examine how 
whisker discrimination is influenced by microstimula- 
tion in $1). 


IR discrimination 

Once an animal reached threshold performance on the 
initial visual discrimination task, we trained them to dis- 
criminate single IR light sources in the same behavioral 
chamber. We first implanted an electrode array into the 
appropriate area (either S1 or V1, as described under 
Surgical procedures). After at least a week of healing from 
surgery, and after determining current thresholds for the 
four stimulating channels (see Microstimulation), we then 
trained them on a task that was structurally identical to the 
initial visual discrimination task. The only difference was 
that we replaced the visual light with IR light. Each rat was 
equipped with four IR sensors distributed evenly around 
the circumference of the animal’s head on the horizontal 
plane (Fig. 18). The IR sensors (Lite-On) had a peak spectral 
sensitivity at a wavelength of 940 wm and a 20° width at half 
of its maximum sensitivity. 

After processing (see IR—Stim transform), information 
from each sensor was projected to a stimulating channel 
at a different cortical location. The information was pro- 
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jected in a topographically natural way: for instance, in- 
formation from the left sensors was projected to the right 
cortical hemispheres, and information from the anterior 
sensor was projected to the anterior region of S1, which 
represents the rostral whiskers (Hartmann et al. 2016). A 
similar pattern was replicated in V1 (Fig. 1B, C). Note also 
that surgeries to implant S1 and V1 were randomly inter- 
leaved, so as not to introduce bias or trends in the pro- 
cedures. 

During training on the IR-discrimination task, the visual 
lights were progressively replaced with IR lights. Specifi- 
cally, they began with a mixture of two types of trials: IR 
only trials, in which only IR lights were turned on, and IR + 
vis trials, in which IR light (and ICMS) was shown, followed 
by visible light, to help animals associate ICMS with rein- 
forced visual light. Specifically, on IR + vis trials, we 
would turn on the IR light, and then 400-500 ms later, turn 
on the visible light, and then 500 ms later, turn off the IR 
light for the rest of the trial. On their first day of training, 
the percentage of IR-only trials was 20%, and this in- 
creased to 100% by the fifth day of training. All percent- 
age correct values on IR discrimination were based on 
IR-only trials. 

A total of 41 rats were trained on the IR discrimination 
task (21 used to compare learning rates on the basic IR 
discrimination task, 6 to look at visual-IR integration, 8 to 
look at whisker discrimination, and 6 trained in the naive 
task; these 6 were not pretrained on the visual task). We 
did not perform a priori power calculations to determine 
the number of animals to use in the study. 


Integrated IR + visual discrimination 

In the integrated IR + visual discrimination task, we 
trained six animals to ignore visual and IR distractors and 
select the port in which both the IR and visual lights were 
activated simultaneously. On each trial, each of the four 
ports was randomly assigned one of four stimulus condi- 
tions: none, IR only, visual only, or both IR and visual (see 
Results). The IR- and visual-only ports were distractors, 
and the animal received a reward only when it ap- 
proached the port with both IR and visual lights on. 


Naive training 

One group of six animals was taught to perform the IR 
discrimination task without being pretrained in the visual 
discrimination task. We first placed these naive rats in the 
behavioral chamber with only a single port placed along 
its circumference, to familiarize them with the behavioral 
chamber. Using the single-port setup, the rat was taught 
to poke a center button to initialize trials and subsequently 
poke the port to receive a reward. To avoid generating a 
directional bias in the rats, we randomly selected a posi- 
tion in the chamber’s circumference where the port would 
be placed every session. Once they were familiarized with 
the chamber (performing 100 trials per session for 3 d), we 
proceeded to implant them with stimulating electrodes 
in either S1 or V1. We then trained the rats using only 
infrared trials (no mixed visual + IR trials as in the stan- 
dard IR discrimination protocol). The trainer was blind to 
the implant location: an assistant was in charge of setting 
a rat in the chamber while the trainer would make adjust- 
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ments to the stimulation currents during the session. Ev- 
ery day, the order of the animals was randomized, and the 
software was written so that the trainer could only adjust 
currents relative to the initial current. 


IR—Stim transform 

The IR to microstimulation frequency transform is a step 
function from IR sensor voltage (which we sometimes refer 
to as “IR level” or “IR intensity”) to microstimulation fre- 
quency as shown in Fig. 1D. The original transform, de- 
scribed in Thomson et al. (2013), was roughly exponential 
in shape, with 17 step intervals. In the previous article, 
researchers used this transform to stimulate rats when 
neuronal activity was not simultaneously recorded. When 
online recording was introduced during the task to limit 
bandwidth, the transform was restricted to only seven 
intervals (Hartmann et al. 2016). 

We used this same transform, but also developed a new 
optimized transform such that each stimulation frequency 
would occur with equal probability over the course of a 
session. To this end, we measured the probability of dif- 
ferent voltage values occurring across multiple sessions. 
We then chose a new transform that would guarantee that 
each voltage step would occur with equal frequency, 
thereby increasing the effective information available to 
the animal on each trial. The voltage steps required to 
guarantee equal likelihood of occurrence were roughly 
logarithmically spaced, and we increased the frequency 
constantly for each step, so the transform from IR level to 
stimulation frequency is logarithmic, as seen in Fig. 1C. It 
is important to mention that we used all frequencies, and 
we observed no harmful effects [we previously have 
avoided frequencies 20-100 Hz (Thomson et al. 2013; 
Hartmann et al. 2016) for fear of inducing kindling (God- 
dard, 1967)]. 


Optimizing the sensitivity of IR sensors 

IR sensors played a crucial role in the IR prosthetic 
system. The IR photodetector circuit included a simple 
voltage divider that let us adjust the sensitivity of the sensor. 
In practice, changing the resistance by 100 kO, decreased 
the measured output of the IR sensor by ~20% at a given IR 
level. This setup allowed us to scale the sensitivity in each IR 
sensor independently. In the IR-visual integration task, 
rats performed the task with their sensitivity set at a high 
value (resistance set to 475 kQ) for several days until 
performance plateaued, usually 75% correct for 5 con- 
secutive days. We then decreased sensitivity in steps, 
allowing the animal to reach a behavioral plateau at each 
new sensitivity level. For each rat, the optimum IR sensi- 
tivity was selected based on the best performance pla- 
teau, calculated over all sensitivity levels tested. 


Aperture-width discrimination 

We trained eight rats to discriminate the width of a 
tunnel using only their facial whiskers, a task discussed 
extensively previously (Krupa et al. 2001, 2004; Wiest 
et al. 2010). More specifically, we trained them to use their 
whiskers to discriminate the size of a variable-width ap- 
erture. Rats were trained to sample the width of a 
variable-width aperture using their whiskers (Fig. 4) and 
were rewarded for moving to the left reward port when the 
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aperture was narrow and to the right reward port when the 
aperture was wide (see Results). Once they reached cri- 
terion performance (80% correct) with widths of 54 and 78 
mm, we then switched some rats to a multiple width 
version (with either 12 or 14 widths). For the multiple width 
task, widths less than 66 mm counted as narrow. 


Surgical procedures 

Surgeries were performed under ketamine (Ketaset) 
and xylazine (AnaSed, Akorn Animal Health) anesthetic 
with 100 mg/kg ketamine and 0.06 mg/kg xylazine. After 
cleaning the surface of the skull, we placed three to six 
titanium screws (Antrin Miniature Specialties) into the 
skull, sealed them in place with Metabond (Parkell), and 
coated the rest of the cleaned skull surface with Meta- 
bond before continuing with the surgery. 

We then performed the craniotomy, removed the dura, 
and implanted electrodes that were built in-house (see 
Electrode design). The coordinates for S1 were -2.5 mm 
posterior and 5.5 mm lateral to bregma. The V1 coordi- 
nates were -7.45 posterior and 3.25 lateral to bregma, 
oriented 30.5° relative to the midline (with the anterior 
portion more lateral; Fig. 1C). After the electrodes were 
lowered to the proper depth (1.5 mm in S1, and either 0.8 
or 1.3 mm in V1; Results), we sealed the craniotomy with 
Quick Set dental acrylic (Coltene) and cyanoacrylic (Hob- 
bylinc.com). Locations of electrodes were verified histo- 
logically (see below) and physiologically by examining 
manual responses to whisker deflections (S1) and LED 
flashes (V1). 


Electrophysiology 
Electrode design 

The stimulating/recording arrays consisted of 32 42-um 
stainless steel microwires arranged into four groups of 
eight (see Fig. 1£; Hartmann et al. 2016). Microwires were 
paired, such that each cortical penetration contained two 
microwires, with the potential for each couple to be used 
as an anodic/cathodic dyad for microstimulation (see Mi- 
crostimulation). The pairs were 300 um apart, and this 
was also the distance between different adjacent wires in 
the same group. Clusters were 2 mm apart within a hemi- 
sphere (Fig. 1£). All electrodes were designed and built 
in-house. 


Histology 

For histologic verification, we used cytochrome oxidase 
(CO) staining of flattened cortical sections, as described 
extensively elsewhere (Wong-Riley, 1979; Hartmann et al. 
2016). Briefly, after perfusing the animal with 0.1 m phos- 
phate buffer, we removed subcortical tissue and flattened 
the cortex overnight in 30% sucrose. After freezing, we 
sliced the brain at 40-60 xm and collected free-sections in 
0.1 m phosphate buffer (pH 7.4) and placed them in standard 
CO reaction solution (Wong-Riley, 1979). We gently agitated 
the slices at room temperature for 2-6 h until the sections 
appeared golden-brown to the eye (Fig. 1F). 


Recording 

Single- and multiunit extracellular neurophysiological 
data were recorded using the Multichannel Acquisition 
Processor (MAP; Plexon). After collecting data, during 


November/December 2017, 4(6) e0262-17.2017 


New Research 5 of 17 


offline sorting we selected spike wave form features (prin- 
cipal components, peak amplitude, etc.) that provided 
optimal separation using Offline Sorter software (Plexon; 
typically we used the principal components, but some- 
times we used other features such as wave form peak or 
wave form energy). Single-unit quality was determined by 
separation of the unit from the noise in the 2- or 3-D 
feature space and the presence of a clear absolute and 
relative refractory period in the autocorrelogram for the 
putative unit. 


Microstimulation 

We stimulated using a four-channel biphasic, charge- 
balanced microstimulator (Hanson et al. 2012). Each stim- 
ulation pulse consisted of a 100-ys pulse from the anodic 
wire, followed by a 50-ys pause, followed by a 100-yus 
pulse in the cathodic wire. 

After the animals recovered from surgery, we deter- 
mined the minimal current threshold for four stimulating 
channels, one in each location. In awake relaxed animals, 
we would deliver a 250-Hz train of stimulation for 75 ms, 
starting with a low current (6 wA), and incrementally in- 
creased the current amplitude until we visually saw a 
reaction from the animal (a movement of the head, loco- 
motion, etc.). We also measured neuronal thresholds: 
we applied 250-Hz trains of stimulation for 75 ms in 
isoflurane-anesthetized animals and noted the current 
amplitude at which we observed a response above base- 
line (the threshold current), and also the amplitude at 
which the neuronal responses saturated (the saturating 
current, which is the current above which we did not 
observe an increase in response). 

In practice, we have found that animals learn the task 
faster when we apply currents well above the threshold 
for evoking a response, as long as the currents are not 
aversive to the animal (Hartmann et al. 2016). Hence, for 
our initial currents for the prosthesis, we typically started 
well above the neuronal threshold, at twice the saturating 
current (typically 50-90 pA). We took it as a sign that a 
current was aversive if the animal began scratching its 
face or stopped participating in the task in response to 
ICMS. 


Whisker receptive field mapping 

To determine the principal whiskers that evoked activity 
in an S1 unit, we recorded in lightly anesthetized (isoflu- 
rane) animals while stimulating individual whiskers with a 
hand-held probe or air puff and logged the whiskers that 
evoked a noticeable change in the baseline response on 
an audio monitor. Note that the goal was to quickly de- 
termine which whiskers needed to be trimmed during the 
whisker discrimination task, not to construct detailed 
quantitative receptive field maps, so we used this rela- 
tively coarse method to minimize the animals’ time under 
general anesthesia. 


Analysis 
Stimulus population vector 

To compactly represent the set of four microstimulation 
bursts delivered to the brain at a given time, we employed 
a stimulus population vector, which is based on popula- 
tion vector representations of movement from the motor 
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Table 1. Statistical analysis 

Figure Test Quantities compared P-value Effect size 
2B Two-tailed t test S1 vs. V1: num sessions to learn 0.0032 1.6259 
2E Two-tailed t test S1 vs. V1: num trials to learn 0.0047 1.6048 
2F ANOVA V1: percent correct (PC) vs. difficulty 0.0015 0.9904 
2F Two-way ANOVA Factor 1 (angle): mean PC 0.0003 0.4261 
2F Two-way ANOVA Factor 2 (implant location): mean PC 0.6186 0.043 
2F Two-way ANOVA Interaction (angle x location): mean PC 0.5024 0.1336 
2G Two-tailed t test Naive V1 vs. pretrained S1: num sessions to learn 0.8518 0.1201 
2G Two-tailed t test Naive S1 vs. V1: num sessions to learn 0.6433 0.4082 
3E F test Stimulus location vs. receptive field peak correlation 0.7 0.0259 
4c Two-tailed t test S1 vs. V1: num sessions to learn integrated task 1 0 

4D Two-tailed t test S1 vs. V1: num trials to learn integrated task 0.6258 0.4306 
SE Two-way ANOVA Factor 1 (animal group): PC for two-width case (54/78) 0.5485 0.0833 
5E Two-way ANOVA Factor 2 (treatment): PC for two-width case 0.0004 1.115 
5E Two-way ANOVA Interaction (group x treatment): PC two widths 0.8597 0.4738 
DE: Two-tailed t test Two-width PC change versus zero: after stimulation 0.0136 1.1567 
5E Two-tailed t test Two-width PC change versus zero: after clipping 0.0014 2.0877 
oF Two-way ANOVA Factor 1 (animal group): PC multiwidth (12/14 widths) 0.9187 0.026 
5F Two-way ANOVA Factor 2 (treatment): PC multiwidth 0.0179 1.1543 
5F Two-way ANOVA Interaction (group X treatment): PC multiwidth 0.6384 0.1217 
5F Two-tailed ¢ test Multiwidth PC change versus zero: after stimulation 0.153 0.6874 
5F Two-tailed ¢ test Multiwidth PC change versus zero: after clipping 0.009 2.1161 
5G Two-way ANOVA Factor 1 (animal group): sensitivity 0.7111 0.0833 
5G Two-way ANOVA Factor 2 (treatment): sensitivity 0.0105 1.115 
5G Two-way ANOVA Interaction (group x treatment): sensitivity 0.0868 0.4738 
5G Two-tailed t test Sensitivity change versus zero: group after stim 0.0107 1.6174 
5G Two-tailed t test Sensitivity change versus zero: group after clip 0.3449 0.4786 
6E Chi-squared test Proportion w/anticipatory response in stim vs. control 0.009 0.18 
6E Chi-squared test Proportion w/response to stim in stim vs. control 0.053 0.13 


control literature (Georgopoulos et al. 1986). Briefly, it 
provides an intuitive geometric 2-D representation of the 
full stimulus being delivered to the four cortical locations 
at a given time. For instance, if the two anterior stimulating 
channels are maximally activated, then the population 
vector points to the front. If the two left channels are 
active, it will point to the left, etc. Mathematically the 
stimulus population vector at time t, s(t) is defined as 


4 


sit) = >) tiv, 


i=1 


where f(t) is the microstimulation frequency in channel 
i, and v; is a vector that points in the direction of IR 
channel i on the rat’s head (for instance, v, = <1,1>: see 
Fig. 1B). For more discussion of the stimulus population 
vector, see Results and Fig. 3. For a more extensive 
introduction, see (Hartmann et al. 2016). 


Psychometric curve fitting 

For the aperture-width discrimination task in which 12 
or 14 widths were presented, we fit the behavioral data to 
a Weibull function: a saturating exponential function with 
four free parameters. We used Matlab to fit our behavioral 
data, in the least squares sense, to the following: 


min + (max min)| 4 e(3)], 


where min is the minimum value, max is the maximum 
value, k is the shape parameter, and A is the scale pa- 
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rameter. This is the standard Weibull curve modified to be 
constrained to have a minimum and maximum value (it 
typically varies between zero and one). 


Statistics 

The different effect size calculations for Table 1 depend 
on the statistic used as described in detail in Cohen (1988) 
(Thompson, 2006; Faul et al. 2007; Olivier and Bell, 2013). 
For t tests, we used Cohen’s d (Cohen, 1988; Faul et al. 
2007). For ANOVA measures, we used Cohen’s f, first 
calculating 7? as the ratio of variance explained SScrect/ 
SS,otar (Thompson, 2006) and then using the standard 
conversion to f (Cohen, 1988; Thompson, 2006): 


Similarly, for regression analysis (which used an F test 
for significance), we defined the effect using the same 
measure, but with the regression coefficient r taking the 
place of 7 (Cohen, 1992; Faul et al. 2007). For chi-squared 
tests, we used Cohen’s o statistic (Cohen, 1992; Faul 
et al. 2007; Olivier and Bell, 2013): 


K 
(Poi ~~ Pei)* 
ue eee, 
i=1 er 


where p,,; is the expected frequency of observations, un- 
der the null hypothesis, to fall in group /, and p,; is the 
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Figure 2. Discriminating IR light with visual cortex. A, Learning curve for IR-discrimination task using V1 (n = 6 animals). Note animals 
typically surpassed criterion (85% correct) within the first day. B, Direct comparison of learning curve for the same task with V1 
implants (blue) and S1 implants (green; n = 15 for S1 implants). C, Same information as in B, but with learning curve superimposed 
when an animal has just a single IR sensor on its head (single-sensor learning data are from a previous study [Thomson et al. 2013]; 
n = 4 animals). D, Trial-based moving average analysis of performance in IR discrimination task. The curve shows percentage correct 
as a function of trial number (moving average of 20 trials) from same data shown in A. E, Same analysis as in C, but with S1-implanted 
animals shown in green for direct comparison. F, Performance in IR-discrimination task as a function of angle 6 between the ports 
(see Fig. 1A). Includes data from V1 and S1 implanted animals with four IR sensors, as well as data from S1-implanted animals with 
a single IR sensor for comparison (blue, green, and red lines respectively with mean + SEM percentage correct). G, Performance of 
naive V1- and S1-implanted animals (mean + SEM percentage correct), that were not pretrained on a visual discrimination task (solid 
lines). Data from S1-implanted animals that were pretrained on a visual discrimination task are included for comparison. There is no 
significant difference, among any of the three groups, in the number of sessions it takes to reach 85% correct in the task (9 > 0.05, 
two-tailed t test). 


observed frequency in group /. All calculations of effect 
size were conducted in Matlab. 


stimulating electrodes bilaterally in V1 and projected in- 
formation from IR sensors to the cortex in a topographi- 
cally natural manner, such that the information from the 
left IR environment was projected to the V1 of the right 
hemisphere (see Methods; Fig. 1B; Hartmann et al. 2016). 

Surprisingly, when IR information was delivered to V1, 
rats learned to discriminate IR sources extremely quickly, 
often within their first day of training (Fig. 2A). This was the 
case in four of six rats. Quantitatively, the V1-implanted 
animals surpassed 85% correct in the IR discrimination 
task in 1.3 + 0.2 d of training (n = 6 animals). This is 
significantly faster than S1-implanted animals, who took 


Results 

To examine the ability of primary visual cortex (V1) to 
absorb a new and otherwise invisible source of electro- 
magnetic signals, we used ICMS to deliver information 
generated by IR sources directly to V1. Rats first learned 
to discriminate visual Cues in a circular behavioral cham- 
ber with four light sources (See Methods; Fig. 1A). We 
then implanted them with an IR prosthesis and trained 
them on an IR discrimination task in which they had to 
correctly identify which of four ports in the behavioral 


chamber had an active IR source (Fig. 1A). 

Infrared information was coupled to V1 via four IR sen- 
sors that were arranged around the circumference of the 
rat’s head, so they were able to perceive a full 360° view 
of the circular arena (see Methods; Fig. 1B). We implanted 
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3.8 + 0.4 d to surpass 85% correct (n = 15 rats; p = 0.003; 
two-tailed t test; Fig. 2B). Note that we have significantly 
more S1-implanted animals because we were able to 
include data from eight animals from a previous study 
(Hartmann et al. 2016). For comparison, Fig. 2C shows 
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the learning curve when information from a single IR 
sensor was projected to S1 (data are from a previous 
study, Thomson et al. 2013). In this case, animals took 
on average 22.3 + 7.6 d to reach 85% correct on the 
task. 

Because the V1-implanted animals were clearly learn- 
ing to perform the IR-discrimination task within the first 
session, we examined how many trials it took them to 
learn the task, using a 20-trial moving average (Fig. 2D). 
When IR information was delivered to V1, it took them on 
average 26.0 + 4.0 IR-only trials to reach 85% correct, 
versus 138.6 + 21.1 IR-only trials when projected to S1 
(Fig. 2E), a significant difference (0 = 0.0047, two-tailed t 
test). During training, such IR-only trials were interleaved 
with mixed trials in which IR light was followed by visible 
light, to facilitate learning the association between IR light 
and reward (see Methods). The total number of trials (both 
IR-only and mixed) required to learn the task was 174.33 + 22 
for V1-implanted animals and 600.2 + 67 for S1- 
implanted animals, a significant difference (0 < 0.001; 
two-tailed t test). 

To examine the spatial acuity of the animals’ ability to 
discriminate IR lights, once they were above criterion at 
the initial task, we varied the angle @ between the ports 
(Fig. 1A). When the ports are placed closer together (i.e., 
6 is decreased), task difficulty is increased because mul- 
tiple IR lights can activate the same IR sensor, thereby 
generating ambiguity in the stimulus. We found that per- 
formance in the task decreased as ports were moved 
together from 90° down to 30°, which is the closest we 
could physically move the ports together in our setup (Fig. 
2F). The decrease was small but statistically significant 
(op = 0.0015; ANOVA). Their performance as a function of 
angle was not significantly different from when implants 
were in S1 (9 = 0.619; two-way ANOVA). 

As a control, we then tested whether V1-stimulated 
animals learned to discriminate IR sources faster because 
they were pretrained in a visual discrimination task that 
was structurally identical to the subsequent IR discrimi- 
nation task. That is, did the V1 animals learn faster be- 
cause they were already using V1 for a visual task, so their 
learning generalized more quickly when microstimulation 
was applied to V1 (perhaps because they were already 
attending to V1)? To test this, we trained six animals on 
the IR discrimination task that had no pretraining on a 
visual discrimination task, three implanted in V1 and three 
in S1. The trainer was blind to the location of implant (see 
Methods). The hypothesis was that if the V1-implanted 
animals learned faster because of visual pretraining, then 
when they did not undergo such training, their perfor- 
mance should converge on that observed in the S1- 
stimulated animals. As can be seen in Fig. 2G, this is 
exactly what we observed. The naive V1-implanted ani- 
mals took 4.0 + 0.57 d to learn the IR discrimination task, 
which was not significantly different from the S1- 
implanted animals pretrained on a visual task, who took 
3.8 + 0.4 d (0 = 0.85; two-tailed t test). The naive S1- 
implanted animals took 4.33 + 0.33 d to learn the task, 
which was also not significantly different from the naive 
V1 animals (0 = 0.64, two-tailed t test). 
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Response to microstimulation in V1 

To help understand how the brain is processing this 
new source of sensory information, we recorded from V1 
neurons in animals that performed well above criterion at 
the task (16 sessions in four animals), using methods 
discussed extensively elsewhere (Hartmann et al. 2016). 
Recording during the task required that we change the 
stimulation protocol. Namely, instead of continuously up- 
dating the stimulating frequency every 50 ms, which pro- 
duces continuous stimulus artifact, during recording we 
would stimulate the brain intermittently. That is, every 140 
ms we would sample the IR levels, and stimulate for just 
75 ms based on those levels, and then for the rest of that 
140-ms period we would turn off the microstimulators, to 
allow for an artifact-free epoch for recording the neuronal 
response during that stimulus cycle. 

To visualize the neuronal response to the distributed IR 
stimulus, we compactly represented the IR-based micro- 
stimulation patterns using a stimulus population vector 
[see Methods; Fig. 3A, B shows how such vectors are 
constructed; see also Hartmann et al. (2016)]. This vector 
provides a compact geometric 2-D representation of the 
full set of four 75-ms ICMS bursts delivered to the cortex 
at a given time. For instance, if the two anterior stimulating 
channels are maximally activated, then the population 
vector points to the front. If the two left channels are 
active, it will point to the left, and so on. 

Fig. 3C shows the set of all population vectors delivered 
during one session in a V1-implanted rat. To show how 
the stimulus changed over time, contiguous stimuli within 
a trial are connected by a line, and later stimuli are indi- 
cated by darker circles in the figure. In a typical trial, the 
stimulating population vector started near the origin at the 
center of the graph, meaning that the microstimulating 
electrodes have very low levels of activation, and moved 
to the top (meaning that the two anterior microstimulating 
electrodes were activated). The mean of the first and last 
three stimulus population vectors are overlaid in red, 
showing that this trend of starting near the origin and 
moving anterior is representative. 

We used peristimulus time histograms (PSTHs) to quantify 
the mean responses of V1 neurons to ICMS during the 
IR-discrimination task. Four representative PSTHs depicting 
the response of V1 neurons to microstimulation during the IR 
task are shown in Fig. 3D. These PSTHs display the mean 
neuronal response to the first and last three microstimulation 
bursts in a trial, averaged over all trials in a session. Note that 
during the time of microstimulation, the PSTH is zero be- 
cause of ICMS-induced stimulus artifact, so each PSTH 
shows responses to microstimulation after this brief artifact 
period. There is a gap between the first and last three ICMS 
bursts to indicate that there were a variable number of 
substimuli on each trial. 

Although they are useful portraits, such PSTHs provide 
a limited description of the full range of neuronal re- 
sponses to ICMS as animals perform the IR discrimination 
task: the first and last three stimuli are incomplete sam- 
ples of the full space of stimuli delivered to the brain 
during the task. To give a more complete picture of neu- 
ronal response, we used the IR receptive field (IR-RF), 
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Figure 3. Neuronal responses to intracortical microstimulation in V1. A, Example of reduction of four-element full representation of 
electrical stimulation vector (four-element vector where each number represents frequency of microstimulation at a different location) 
down to two-dimensional population vector. There are five examples from five different time points T1-T5. B, Geometric represen- 
tation of the population vectors from A. C, Stimulus population vectors from all trials in a session, with mean of first and last three 
substimuli overlaid in red. The black diamond outline is the convex hull of stimulus population vectors, showing the perimeter of the 
set of all possible vectors. Note that only a fraction of this set is actually presented in a given behavioral session. See text for more 
details. D, Neuronal data from four representative units. PSTHs show mean response to first three and last three microstimulation 
pulses, averaged over all trials in the session (note there are different axis limits for the different PSTHs). Associated with each PSTH 
is an IR-RF, a contour plot that shows the mean number of spikes in response to each stimulus population vector, with hot colors 
representing high firing rates. E, Distribution of all 240 IR-RF centers from all units recorded: note they tend to concentrate at the two 
medial corners of the stimulus space, although these are fairly rare events (C). F, Polar count histogram illustrates the medial, rather 
than anterior, concentration of IR-RF angular distribution. G, H, Same as E and F, but from data from S1 from a previous study 


(Hartmann et al. 2016). Note the anterior distribution of the IR-RF centers, much closer to the distribution of the stimuli. 


which measures the mean spike count as a function of 
stimulus population vector, with the count calculated for 
all population vectors delivered during a session. Fig. 3D 
shows the IR-RFs for four neurons, adjacent to their 
PSTHs. For instance, unit 28 preferred IR stimuli that were 
presented to the left of the animal. This explains why its 
firing responses to the first and last three stimuli, shown in 
the PSTH, were relatively small. 

This lateralization of IR-RF centers was typical in V1, as 
can be seen in Fig. 3E, which shows a contour plot 
depicting the distribution of IR-RF peaks for all 240 V1 
units recorded in this study. There was a pronounced 
trend for these V1 IR-RF peaks to be lateralized to the left 
and right sides. This is more clearly demonstrated by the 
polar count histogram of receptive field peak angular 
location in Fig. 3F. This is in sharp contrast to what was 
previously observed in S1, in which the IR-RF centers 
tended to closely match the stimulus statistics: that is, the 
S1 neurons had a strong preference for stimuli in the 
anteromedial sector of the stimulus space, as shown in 
Figs. 3G and 3H, which are reproduced from previous 
data (Hartmann et al. 2016). 

Although previous research demonstrated the exis- 
tence of a highly significant 2D correlation between IR-RF 
peak location and stimulus location in S1 (Hartmann et al. 
2016), in V1 there was no such match between IR-RF 
peak and stimulus location: the 2-D correlation between 
IR stimulus distribution (the set of all stimulus population 
vectors) and IR-RF center was weak and not significant 
(0 = -0.03; p = 0.7; F test). 


Integration of visual and IR information in V1 

When using the IR prosthesis, both native (visual) and 
novel (IR) information streams were projected to the same 
primary sensory area. To determine whether rats could 
learn to simultaneously integrate visual and IR information 
within V1, we trained them on a visual-IR integration task. 
During this task, we presented multiple visual and IR lights 
in the behavior chamber simultaneously. To receive a 
reward, rats had to select the single port in which both 
visible and IR lights were active (see Methods). On each 
trial, four stimuli were presented in the standard cylindri- 
cal behavioral chamber (Fig. 4A): in the target port, both 
the visual and IR lights were activated simultaneously. In 
another, a visual distractor light was turned on, while a 
third port contained an IR distractor light. The fourth port 
showed neither light. The location of each of these ports 
was selected randomly on each trial. 
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In preliminary testing, one cohort of three animals had 
trouble learning this task with high accuracy, often going 
to the first port that they noticed with an IR light activated. 
This seemed to be partly because they were being over- 
stimulated in the relatively small chamber in which they 
were initially trained. To test this hypothesis, we trained a 
second cohort of three animals in a larger-diameter be- 
havior chamber (20-inch or ~0.5-m diameter), and signif- 
icantly decreased the sensitivity of the IR sensors, so 
when a trial was initiated in the middle of the chamber, the 
IR sensors were not as likely to trigger ICMS at the start of 
the trial. Further, we optimized the transform from IR 
sensor output to ICMS frequency so that no frequencies 
were oversampled (see Methods; Fig. 1D). 

After such optimizations were in place, animals readily 
learned to perform the visual-IR integration task, reaching 
85% correct within 3.67 + 0.67 sessions (n = 3 animals; 
Fig. 4B). Despite this performance, we noticed that some 
of the animals tended to make more errors toward the IR 
or visual distractor and hypothesized that they might be 
biased by microstimulation frequency. Hence, we system- 
atically varied the maximum frequency of stimulation until 
we found the frequency that optimized percentage correct in 
the task (see Methods). Before frequency optimization, the 
best performance over five sessions was 82.7 + 1.0%, and 
after optimizing frequency, the best performance over five 
sessions was 93.5 + 1.4% (Fig. 4C). During initial training, 
we stimulated at a maximum frequency of 300 Hz and 
found that the optimal frequency, that minimized errors, 
was 133 + 33 Hz. 

We next tested the hypothesis that processing both IR 
and visual inputs in V1 would interfere with performance 
of the visual-IR integration task. In particular, we im- 
planted a group of three animals with stimulating elec- 
trodes in S1 and trained them in the visual-IR integration 
task. In these animals, the IR information would be pro- 
cessed in S1, while the visual information would go to V1, 
so direct sensory interference between the two types of 
information would not occur. Yet, animals would still have 
to directly compare both sources of sensory informa- 
tion—IR light coming through $1 and visible light through 
V1i—to solve the task properly. Hence, if the S1-implanted 
animals performed better on the visual-IR integration task, 
this would provide an indirect measure of sensory inter- 
ference within V1 during the sensory integration task, 
while any residual errors could be interpreted as a mea- 
sure of intrinsic task difficulty. 
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Figure 4. Visual-IR integration task. A, Schematic of the task. In a cylindrical chamber with the same ports shown in Fig. 1A, four 
stimuli are randomly assigned to the ports. The target stimulus is the one with both visual and IR lights on. The IR distractor has only 
an IR light on, the visual distractor only a visual light, and there is one port with no light activated. B, Learning curve from three animals 
trained on the visual-IR integration task (implants in V1). The discontinuous right portion of the curve shows the last four sessions 
using an optimized frequency (see text). C, Comparison of learning curves between V1- and S1-implanted rats in the visual-IR 
integration task (n = 3 animals for each group). D, Same data as in C, but with trial-based moving average (moving window average 


taken with 20 trials). 


Surprisingly, the S1-implanted animals reached 85% 
correct within 3.67 + 0.88 sessions (Fig. 4C), which was 
not significantly different from the V1-implanted animals 
(op > 0.5, two-tailed t test). When we analyzed the results 
by trial, we saw similar results (Fig. 4D). The S1-implanted 
animals reached 85% correct within 139 + 51 trials, and 
the V1 implanted animals reached 85% correct within 103 + 
45 trials, and these learning rates were not statistically 
different (0 = 0.626; two-tailed t test). These results sug- 
gest that ICMS in V1 does not appreciably impair the 
ability of V1 to process incoming visual stimuli while 
simultaneously receiving native visual information. More- 
over, it demonstrates that rats can combine invisible (IR) 
and visible light, delivered to two different primary cortical 
areas (S1 and V1), as efficiently as when both sensory 
signals converge in their V1, to solve this difficult sensory 
integration task. 


Effect of prosthesis use on native tactile processing 
In a final set of experiments, we further explored the 
consequences, for the native sensory modality, of project- 
ing new information to a primary sensory area. Previous 
research has suggested that sensory prosthetic systems 
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that employ ICMS might compromise the original function 
of the cortical region to which information is projected (Ni 
and Maunsell, 2010). Here, we measured whether rats in 
which IR information was projected to the whisker repre- 
sentation in S1 showed any decline in their ability to 
perform a whisker-dependent tactile discrimination task. 
Toward this end, we trained animals in two behavioral 
tasks each day: the stimulation-based IR discrimination 
task and a whisker-dependent aperture-width discrimina- 
tion task (see Methods; Fig. 5A; Krupa et al. 2001, 2004; 
Thomson et al. 2014). 

Briefly, in the aperture-width discrimination task, a 
variable-width aperture is moved to one of two widths (54 
or 78 mm), and after sampling the aperture with its large 
facial whiskers, the rat must select the correct associated 
reward port to receive reward (Fig. 5A). We trained eight 
rats on this two-width discrimination task, and trained six 
of these rats on a multiwidth version of the task in which 
we randomly selected from more than just two widths 
(either 12 or 14). Widths >66 mm were considered wide, 
and those narrower than 66 mm were considered narrow. 
This allowed us to measure the bias (preference for one 
width over another), sensitivity (maximum slope of the 
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Figure 5. Effects of S1 prosthesis on whisker discrimination. A, Aperture-width discrimination task. After the movable bars making 
up aperture move to the desired position, the rat enters the stimulus chamber, sweeps its whiskers across the aperture (movable bars 
controlled by the computer), and activates the central nose poke (CNP). The animal then retreats into the stimulus chamber and pokes 
in the right or left reward port depending on the aperture width. B, Psychometric curve from a single session in the 14-width 
aperture-width discrimination task. The points are the mean proportion of left responses at each width; the red line is the best 
minimum least-squares fit to the points, fitted using a Weibull function. C, As in B, but in an animal with worse performance (lower 
maximum slope of the curve and lower peak performance at each end of the spectrum). D, Performance of stimulated and matched 
control animal in the whisker discrimination task during different stages of the experiment. Before stimulation (Baseline), while learning 
the IR discrimination task via ICMS (Stimulation), and after whiskers were clipped (Clipped). See text for details. E, Mean + SEM 
change in performance after the two transitions illustrated in D. AStim, transition from baseline to stimulation; AClip, transition from 
normal stimulation to stimulation with whiskers clipped. This analysis is restricted only to 54/78-mm widths, and includes eight 
animals (four stimulated/four control for the Astim transition) and seven animals (four stimulated/three control) for the Aclip transition 
(one control animal stopped performing the task after whisker clipping). F, Same as E, but for the full multiple-aperture with 
discrimination task (either 12 or 14 widths). This includes six animals for the first transition and five animals for the second (as before, 
one control animal stopped behaving after whisker clipping). Percentage correct is calculated as the integral under the best Weibull 
fit to the psychometric curve (see Methods). G, Change in behavioral sensitivity, or the measure of the maximum slope of the best 
fit curves to the psychometric data, with each transition in the task. Lower slope means less behavioral sensitivity. 


psychometric curve), and standard overall percentage task, before any S1 microstimulation. The second, stim- 
correct on the task over the full range of widths presented _—_ulation, phase shows performance on the aperture-width 
(Fig.5B8;.G): discrimination task after the stimulated group began training 

Once rats were trained on the two tasks, we implanted —_ on the IR discrimination task. The third and final whisker clip 
them with stimulating electrodes in S1 as before. We phase was when all whiskers, except those corresponding 
trained four rats to perform stimulation-based IR discrim- to the barrels being stimulated, were clipped on each side 
ination (the stimulated group) and four to perform the _ ofthe face. It is important to note that at each phase, each 
equivalent visual discrimination task as described above __ rat was performing two tasks each day: the whisker dis- 
(the control group). This allowed us to track tactile dis- | crimination task and either the IR discrimination task 
crimination performance in the two groups, each perform- (stimulated group) or visual discrimination task (control 
ing two similar tasks a day, with only one group receiving group). Fig. 5D plots their performance only in the 
microstimulation in S1. aperture-width discrimination task. 

Fig. 5D shows the raw performance (percentage cor- The third, whisker clipping, phase was crucial because 
rect) in the multiwidth discrimination task in three key _ it is Known that rats can perform the aperture-width dis- 
phases, in both the stimulated and control groups. The — crimination task above chance with just one whisker re- 
first, baseline, phase shows the percentage correct in| maining on each side of the face (Krupa et al. 2001). 
both groups on the initial four-choice visual discrimination | Hence, to control for the possibility that the stimulated 
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group could perform the task using the whiskers that were 
outside of the region being stimulated in S1, we clipped all 
of the facial whiskers that did not correspond to barrels 
that were stimulated by the prosthesis (stimulated group) 
or simply whiskers outside of the region of the implant 
(control group). After determining the whiskers that evoked 
activity at the site of S1 stimulation, we clipped the remain- 
ing whiskers on the face of the rat. This meant we clipped 
~20 whiskers, leaving only eight whiskers on each side of 
the face. 

To determine whether ICMS, or trimming whiskers, al- 
tered the performance of the stimulated relative to control 
animals in the aperture-width discrimination task, we per- 
formed a 2 X 2 ANOVA [factor 1: animal group (control/ 
stimulated); factor 2: treatment (stimulation/whisker 
clipping)]. Crucially, there was no significant difference in 
performance change between the two groups of animals 
using any of the measures, but there was a significant 
effect of treatment using each measure (a = 0.05: Figs. 
5E-G; Table 1). 

For instance, Fig. 5 shows the mean response change 
for all animals in the 54 vs. 78 whisker discrimination task, 
both when they were switched from the baseline to the 
stimulation phase (AStim) and when they were switched 
from the stimulation phase to the whisker clip phase 
(AClip). Fig. 5F shows a similar plot, but for percentage 
correct calculated using data from the multiwidth version 
of the task. Fig. 5G examines changes in sensitivity (the 
slope of the psychometric curve). Using all of these mea- 
sures, there was no significant difference between 
groups. This suggests that microstimulation did not ap- 
preciably impair the ability of the stimulated rats to use 
their whiskers in the aperture-width discrimination task. 

However, there were significant effects of treatment in 
each case (Table 1). Because the changes were similar in 
the two groups between phases of the task, to examine 
the sources of the differences we lumped the data from 
the stimulated and control groups together and looked for 
potential effects of each change in condition in a post hoc 
analysis. Surprisingly, percentage correct actually in- 
creased between baseline and stimulation phases, sug- 
gesting an effect of continued learning (Fig. 5E, F). This 
was significant in the eight animals trained on the two- 
width task (9 = 0.01; two-tailed t test), but not significant 
in the multiwidth task (0 = 0.153). Not surprisingly, the 
performance drop was significant after whisker clipping 
(op = 0.001 for the 54/78 task; p = 0.009 for the multiwidth 
task; two-tailed t test). The animals’ sensitivity increased 
during the stimulation period (a significant change: p = 
0.01; two-tailed t test), and dropped after whisker clip- 
ping, although this drop was not significant (9 = 0.34). 
Surprisingly, using all of these measures, the performance 
drop after whisker clipping was actually less for the stim- 
ulated group (Fig. 5E—G), but this trend was not significant 
(a = 0.05). 

We next examined neuronal activity in S1 during per- 
formance of the aperture-width discrimination task in the 
stimulated and control groups. It is Known that S1 neu- 
rons in animals trained to use S1 in an ICMS-based IR 
discrimination task maintain their ability to respond to 
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whisker deflection outside of a task context (Thomson 
et al. 2013; Hartmann et al. 2016). Here, we recorded from 
populations of S1 neurons as two stimulated and two 
control animals performed the two-width aperture-width 
discrimination task. Fig. 6A, B shows sample PSTHs and 
the set of all PSTHs from all 104 S1 single units from the 
control animals, respectively. Fig. 6C, D shows the same 
for the 117 S1 units in the stimulated animals. 

Overall 87% of neurons in the control group and 86% of 
those in the stimulated group displayed a significant re- 
sponse at some point in the trial. However, there were 
some quantitative differences in the two groups of ani- 
mals. As has been reported extensively before, S1 neu- 
ronal activity during the aperture-width discrimination 
task can be analyzed in different time epochs around the 
instant of whisker contact with the port edges (Krupa et al. 
2004; Pais-Vieira et al. 2013; Thomson et al. 2014). For 
instance, anticipatory activity in S1 has been commonly 
observed during the 400 ms that precedes the contact 
between the whiskers and the aperture (Krupa et al. 2004; 
Wiest et al. 2010; Pais-Vieira et al. 2013). We observed 
such anticipatory activity here as well, and the proportion 
of neurons showing such responses was larger in the 
stimulated group (0.71) than the control group (0.51; p = 
0.009; chi-squared test for proportions). Further, the pro- 
portion of neurons responding to whisker deflections in 
the stimulus epoch (the period 350 ms after whisker con- 
tact with the aperture) was larger in the stimulated group 
than the control group (0.84 vs. 0.73), although this dif- 
ference was not statistically significant (0 = 0.053; chi- 
squared test). 


Discussion 

We demonstrated that primary sensory areas in adult 
rats exhibit a noteworthy ability to absorb novel sources 
of information without compromising preexisting sensory 
function. It seems that primary cortical sensory areas in 
mammals can rapidly integrate two independent sensory 
streams while exhibiting very little large-scale interfer- 
ence. As was pointed out previously (Hartmann et al. 
2016), the fact that the novel information source was IR 
light was arbitrary; other information sources, such as 
X-rays or microwave radiation, could be used if an appro- 
priate portable sensor were available. 


Comparative analysis: accelerated learning with 
area-specific pretraining 

This study is a first step in a comparative neuronal, 
behavioral, and functional analysis of the absorption of 
novel sources of sensory information in multiple primary 
sensory areas (Yang and Zador, 2012). Based on the 
anatomic and functional similarities in different sensory 
areas (Miller, 2016), and the fact that diverse areas can so 
readily acquire new functions in juveniles (Sur et al. 1988), 
we expected both V1 and S1 to exhibit the same proper- 
ties when presented with IR information. 

This is the result we observed, as rats implanted with 
S1 and V1 prosthetic devices both learned at the same 
rate (Fig. 2F). However, when pretrained on a structurally 
similar visual discrimination task, significant differences 
emerged between the two populations: the V1-implanted 
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Figure 6. Neuronal responses during the aperture-discrimination task. A, Four example PSTHs during the aperture-width discrimi- 
nation task described in Fig. 5A, from control animals that were not stimulated. Bin widths are 50 ms. Reference lines show mean + 
SD during the baseline period (the period before -400 ms). Time 0 is the time of stimulus onset, when the whiskers contact the 
aperture. B, Set of all 104 PSTHs from two control animals. d, door opens to start trial; s, stimulus onset (whiskers contact aperture); 
t, tone sounds indicating rat has reached end of stimulus chamber. Spike counts in B are calculated by subtracting the mean spike 
count during the baseline period. PSTHs below the gray line are those for which there was no significant response. C, D, Same as 
A and B but for the stimulated animals (n = 117 neurons). E, Mean + SEM PSTH for all neurons for control (red) and stimulated (black) 


groups. 


animals learned the task significantly faster than those 
implanted in S1 (Fig. 2). Typically, after pretraining on a 
visual discrimination task, V1-implanted rats learned the 
corresponding IR discrimination task during their first day 
of training, several days and hundreds of trials ahead of 
their S1-implanted counterparts. 

When this visual pretraining was removed, this differ- 
ence in learning rates between S1 and V1 learning rates 
disappeared (Fig. 2F), suggesting that there is no intrinsic 
difference in learning rates between the two popula- 
tions. Cumulatively, these data support our initial hy- 
pothesis that there is an equipotentiality of plasticity 
across adult primary sensory areas: they are equally 
capable of absorbing a new source of distal sensory 
information and using it to perform a sensory discrim- 
ination task. 
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However, when pretrained on a structurally similar task 
that relies on a particular primary cortical area, there was 
a much more rapid transfer of learning when the new 
information was projected to that same primary cortical 
area (Fig. 2A, £). This suggests there exists a form of 
metaplasticity, in which the V1-implanted animals have 
learned to learn the task, transferring their earlier training 
to the new task. Such phenomena have been observed in 
the whisker system (Harris et al. 1999) and in cross-modal 
learning transfer (Over and Mackintosh, 1969). We do not 
yet know the underlying reason for this rapid transfer of 
learning, but speculate that during the visual task atten- 
tion is already directed to V1, so the process of extracting 
and using the information from the prosthesis is facilitated 
in the V1-implanted animals. One way to test this hypoth- 
esis would be to pretrain the animals on a structurally 
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similar tactile discrimination task and see whether the 
S1-implanted animals learn the IR-discrimination task 
faster than the V1-implanted animals. 

Originally, it seemed a natural hypothesis that the V1- 
implanted animals learned faster because V1 is simply a 
more natural home for IR information: IR light is a distal 
electromagnetic cue, which V1 already processes, so 
there is no need to adapt new behavioral strategies for 
processing information as there is in S1. However, the fact 
that S1 and V1 learning rates converge when the animals 
are not pretrained on a visual discrimination task suggests 
there is no such intrinsic preference for IR information in 
V1 compared with S1. 


Multimodal integration within and across primary 
sensory areas 

With the visual-IR integration task used in this study, 
information from both the visual and IR modalities had to 
be integrated and compared for rats to successfully com- 
plete a trial. If the animal solely focused on one type of 
information, they would incorrectly approach a distractor 
port (Fig. 3A). This is typically what happened at first, but 
within four sessions, subjects learned to approach the 
correct port (Fig. 3B). In the future, we will merge such 
multimodal stimuli in different combinations and intensi- 
ties to more thoroughly quantify how animals integrate IR 
and visual information, in particular to measure how 
closely they approach the theoretical Bayesian optimum, 
as when variability is added to one of the cues, making it 
less reliable than the other (Deneve and Pouget, 2004; 
Dadarlat et al. 2015). 

Surprisingly, there was no significant difference in per- 
formance in the visual-IR integration task when the IR 
information was projected to S1 or V1 (Fig. 3C). We had 
expected the S1-implanted animals to learn faster be- 
cause there would be no intra-area sensory interference 
between the visual and IR modalities. Instead, our findings 
suggest that it may not matter whether the two informa- 
tion sources are projected to the same or different cortical 
areas. 

The limits of such multimodal sensory augmentation are 
unknown. How many different types of information can be 
projected to one sensory area before the animal becomes 
confused? This question could have direct clinical rele- 
vance: in designing closed-loop feedback systems for 
artificial limbs, researchers aim to provide subjects with 
multiple types of sensory information, such as tempera- 
ture, pressure, and proprioception (Donati et al. 2016). 
Hence, it will be important to determine the practical limits 
of augmented brain function. 


Conservation of native sensory function 

It has been shown that when IR information is projected 
to the whisker region of S1, S1 neurons still show robust 
response to whisker deflections (Thomson et al. 2013; Hart- 
mann et al. 2016). However, it was not known whether this 
cortical sensory neuroprosthesis would impair the native 
sensory function of the target cortical area. Would S1 
become effectively hijacked by ICMS, and less capable of 
processing tactile information (Griffin et al. 2011)? There is 
evidence from monkeys that learning to perform ICMS- 
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based threshold detection tasks in V1 impairs perfor- 
mance on visual detection tasks (Ni and Maunsell, 2010). 
Contrary to such findings, when we trained rats with the 
ICMS-based prosthesis on a whisker-based tactile dis- 
crimination task, we found no decline in the animals’ 
tactile discrimination (Fig. 5). Therefore, our results 
suggest that projecting novel sensory information to a 
primary sensory cortical area does not necessarily com- 
promise the ability of the animal to use that area for 
sensory discrimination. 

There are multiple differences between our paradigm 
and those in the previous study that could explain the 
different results. The previous research used a detection 
task in fixated, head-fixed animals and purposely limited 
the cue to a small region of visual space (Ni and Maunsell, 
2010). By comparison, we used a discrimination task in 
which freely moving animals swept multiple whiskers across 
the target stimulus: this stimulus is Known to activate a 
broadly distributed population of neurons across the trigem- 
inal system (Krupa et al. 2004; Ferezou et al. 2006; 
Pais-Vieira et al. 2013, 2015). Similarly, the IR prosthesis 
was used by freely moving animals, with information dis- 
tributed to four locations, in animals trained to forage for 
IR information in their environment over extended time 
scales, rather than a single 250-ms microstimulation train 
localized to a single location, as in the previous study (Ni 
and Maunsell, 2010). Finally, detection and discrimination 
tasks are quite different tasks, likely recruiting different 
underlying brain states optimized to their particular goal 
(Sherman, 2001). For instance, when performing a detec- 
tion task, neurons in the cortex and thalamus may tend to 
fire in bursts, which is optimal for indicating that an event 
has occurred, but not for tracking its fine-grained fea- 
tures. When in discrimination mode, thalamic neurons 
exhibit more tonic background firing rates, and may track 
features of the environment at a finer grain and respond 
with lower magnitudes than in detection mode (Fanselow 
and Nicolelis, 1999; Adibi and Arabzadeh, 2011; Olleren- 
shaw et al. 2014). 

Overall, although ICMS use may selectively increase 
localized detection thresholds (Ni and Maunsell, 2010), 
this does not mean that it harms performance in discrim- 
ination tasks in freely moving animals actively extracting 
sensory information from their environment. 

In conclusion, our findings demonstrate that the mam- 
malian brain remains extremely sensitive to the statistical 
structure of the inputs it receives throughout adulthood. 
This has long been clear from studies of newborns (Frost 
and Metin, 1985; Sur et al. 1988) and deafferented adults 
(Kaas et al. 1983; Kossut et al. 1988; Nicolelis et al. 1993). 
The current study shows that it is possible to superimpose, 
without any deafferentation, multiple streams of information 
simultaneously onto the same primary cortical area, whether 
it be S1 or V1. In this context, the present study strongly 
suggests that the upper limits of plasticity of a primary 
sensory cortical area, as well as its ability to adaptively 
absorb and use multiple sources of information, are much 
greater than previously anticipated. This is auspicious for 
the development of sensory cortical prosthetic systems, 
in particular those that might require the use of sensory 
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substitution systems, such as those in which visual infor- 
mation is projected to the somatosensory system (Bach-y- 
Rita et al. 1969). 
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A brain-to-brain interface (BTBI) enabled a real-time transfer of behaviorally meaningful sensorimotor 
information between the brains of two rats. In this BTBI, an “encoder” rat performed sensorimotor tasks 
that required it to select from two choices of tactile or visual stimuli. While the encoder rat performed the 
task, samples of its cortical activity were transmitted to matching cortical areas of a “decoder” rat using 
intracortical microstimulation (ICMS). The decoder rat learned to make similar behavioral selections, 
guided solely by the information provided by the encoder rat’s brain. These results demonstrated that a 
complex system was formed by coupling the animals’ brains, suggesting that BTBIs can enable dyads or 
networks of animal’s brains to exchange, process, and store information and, hence, serve as the basis for 
studies of novel types of social interaction and for biological computing devices. 


} n his seminal study on information transfer between biological organisms, Ralph Hartley wrote that “in any 
given communication the sender mentally selects a particular symbol and by some bodily motion, as his vocal 
| mechanism, causes the receiver to be directed to that particular symbol”’. Brain-machine interfaces (BMIs) have 
emerged as a new paradigm that allows brain-derived information to control artificial actuators* and commun- 
icate the subject’s motor intention to the outside world without the interference of the subject’s body. For the past 
decade and a half, numerous studies have shown how brain-derived motor signals can be utilized to control the 
movements of a variety of mechanical, electronic and even virtual external devices**. Recently, intracortical 
microstimulation (ICMS) has been added to the classical BMI paradigm to allow artificial sensory feedback 
signals”*, generated by these brain-controlled actuators, to be delivered back to the subject’s brain simultaneously 
with the extraction of cortical motor commands””. 

In the present study, we took the BMI approach to a new direction altogether and tested whether it could be 
employed to establish a new artificial communication channel between animals; one capable of transmitting 
behaviorally relevant sensorimotor information in real-time between two brains that, for all purposes, would 
from now on act together towards the fulfillment of a particular behavioral task. Previously, we have reported that 
specific motor’”” and sensory parameters’*"* can be extracted from populations of cortical neurons using linear 
or nonlinear decoders in real-time. Here, we tested the hypothesis that a similar decoding performed by a 
“recipient brain” was sufficient to guide behavioral responses in sensorimotor tasks, therefore constituting a 
Brain-to-Brain Interface (BTBI)’* (Figure 1). To test this hypothesis, we conducted three experiments in which 
different patterns of cortical sensorimotor signals, coding a particular behavioral response, were recorded in one 
rat (heretofore named the “encoder” rat) and then transmitted directly to the brain of another animal (i.e. the 
“decoder” rat), via intra-cortical microstimulation (ICMS). All BTBI experiments described below were con- 
ducted in awake, behaving rats chronically implanted with cortical microelectrode arrays capable of both neur- 
onal ensemble recordings and intracortical microstimulation'’. We demonstrated that pairs of rats could 
cooperate through a BTBI to achieve a common behavioral goal. 


Results 

In our training paradigm, animals learned basic elements of the tasks prior to participating in any BTBI experi- 
ments. First, prospective encoder rats were trained to respond to either tactile or visual stimuli until they reached 
95% correct trials accuracy. Meanwhile, decoder rats were trained to become proficient while receiving ICMS as a 
stimulus. A train of ICMS pulses instructed the animal to select one of the levers/nose pokes, whereas a single 
ICMS pulse instructed a response to the other option. Decoder rats reached a 78.77% + 2.1 correct trials 
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Figure 1 | Experimental apparatus scheme of a BTBI for transferring cortical motor signals. Arrows represent the flow of information from the 

encoder to the decoder rat. In the motor task, the encoder rat has to identify a visual stimulus, signaled by an LED (red circle), and then press one of two 
levers to receive a small water reward. Meanwhile, M1 neural activity is recorded from the encoder rat and transmitted to the decoder animal, by 

comparing the pattern of the encoder’s M1 to a template trial (previously built with the firing rate average of a trial sample). The difference between the 
number of spikes in a given trial and the template trial is used to calculate a Zscore. The Zscore is then converted, through a sigmoid function centered on 
the mean of the template trial, into an ICMS pattern. Thus, the microstimulation patterns varied in real time, according to the number of spikes recorded 
from the encoder rat’s M1, ona trial by trial basis. Once microstimulation is delivered to the M1 cortex of the decoder rat, this animal has to select the same 
lever pressed by the encoder. Notice that the correct lever to press is cued only by the pattern of the decoder’s M1 microstimulation. If the decoder rat 
pressed the correct lever, both rats were rewarded. Thus, when the information transfer between the brains of the two rats was successful, the encoder rat 


received an additional reward that served as positive reinforcement. 


performance level. After this preliminary training was completed, the 
animals were run in pairs, each one in a separate operant box. 

The next phase of training began with the encoder rat performing 
~10 trials of the motor or tactile task, which were used to construct a 
cortical ensemble template, i.e. the mean cortical neuronal activity 
for one of the responses. The increased firing rate associated with the 
right lever press was selected as the parameter extracted from the 
neuronal ensemble in the motor task. The increased firing rate assoc- 
iated with the “Narrow” trials was selected as the parameter extracted 
from the neuronal ensemble in the tactile task. A BTBI session fol- 
lowed in which ICMS trains applied to the cortex of the decoder rat 
reflected the difference between the template and single-trial neur- 
onal ensemble rates produced by a sample of the encoder rat’s M1 or 
S1 activity. ICMS duration (i.e. number of pulses delivered) was 
proportional to the difference between the sampled neuronal 
ensemble firing rate recorded during a given trial and the template 
normalized by the standard deviation. The time window for the 
analysis of neuronal activity and ICMS parameters was adjusted in 
each recording session to maximize the directional signal. The 
decoder rats reacted to ICMS patterns and not any other cues, as 
was evident from control experiments in which the performance of 
those rats dropped to chance level after the ICMS cable was discon- 
nected from the stimulator. Furthermore, the encoder rat received 
feedback information describing the single trial performance of the 
decoder rat; each time the decoder rat responded correctly to the 
ICMS cue, the encoder rat received an additional reward (i.e. water). 

In experiment 1 (Figure 1), encoder rats (N = 3) pressed one of 
two levers after an LED on top of the lever was turned on. While the 
rats did so, M1 neuronal activity was recorded, compared to the 
template and transformed into ICMS trains applied to M1 of the 
decoder rats (N = 4) who performed the same lever press task. As 
would be expected, the encoder rats performed better (95.87% + 1.83 


correct trials) (Figure 2 A) than the decoder rats (64.32 + 1.1%; 
range: 60 - 72% correct trials; Binomial test: P < 0.05 in all sessions) 
(Figure 2 A and B). Yet, the performance of the decoder animals was 
above chance and highly significant. Indeed, in some experiments the 
decoder rat’s performance using the BTBI was very close to the 
maximum performance obtained when ICMS was used alone in 
these animals (72% BTBI vs 78% ICMS alone, see above). 

The primary factor that influenced the decoder rat’s performance 
was the quality of spatial information extracted from the encoder 
rat’s M1. The performance was high if the chosen neuronal ensemble 
accurately encoded left versus right presses (Figure 2 C and Figure 3 
A-D). The higher the deviation from the template and hence the 
larger the duration of the ICMS (i.e. number of pulses delivered), the 
better was the decoder rat’s performance (Figure 2 C and 3 B-D). For 
this first experiment, a total of 538 units and 110 multiunits were 
recorded from encoder rats. Sessions were comprised of 48.04 + 
1.5 trials. The response latency of the encoder rats was 20.06 + 
1.0 seconds, while decoder animals responded at 13.59 + 0.5 sec- 
onds. Note that this difference reflects only the effect of both rats 
working as a dyad (see below for comparison of latencies during 
training and testing). 

In addition to the neuronal transfer from the encoder to the 
decoder rats, feedback information, related to the decoder rats’ per- 
formance, was sent back to the encoder animal. This feedback pro- 
vided an additional reward to the encoder rat every time the decoder 
rat performed a trial correctly. Under these conditions, the encoder 
rats’ response latency decreased after the decoder rat made an error 
(after correct response: 20.67 + 1.665 seconds and after an incorrect 
response = 15.26 + 2.031 seconds; Mann Whitney U = 13570; P< 
0.0001). Furthermore, an analysis of the variation in Z-scores, 
demonstrated that the signal to noise ratio of the neural activity 
extracted from the encoder rat’s M1 increased after the decoder rat 
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Figure 2 | Behavioral performance using a BTBI for transferring cortical 
motor signals. A) Performance of encoder and decoder animals during 
transfer of motor information via a BIBI. The performance of the encoder 
animals was above 90% in all but one session. The BTBI allowed the 
decoder animals to repeatedly perform significantly above chance. This 
performance immediately dropped to chance levels when the cable was 
disconnected but the system remained fully functional. B) The 
performance of the decoder animals across a session is presented with a 
moving average of 10 trials. C) The panel depicts the fraction of right lever 
presses after different microstimulation patterns were delivered to the 
decoder’s M1. As the number of microstimulation pulses increased, a 
higher fraction of right lever presses occurred. The microstimulation 
threshold for response in most animals was situated between 41 and 60 
pulses. 


committed an error (Chi Square = 4.08, df=1; P = 0.0434). Thus, 
both the behavior and neuronal modulations of the encoder rat 
became dependent on the trial by trial behavioral performance of 
its dyad partner, the decoder rat. 

In experiment 2, we tested whether a BTBI could enable a real- 
time transfer of tactile information between a pair of rats’ brains 


(Figure 4). Encoder rats (N = 2) were trained to discriminate the 
diameter of an aperture width with their whiskers’. If the aperture 
was narrow, rats were required to nose poke on the left side of the 
chamber, otherwise they had to poke on the right side of the cham- 
ber. Decoder rats (N = 5) were trained to poke on the left water port 
(narrow aperture) in the presence of ICMS and on the right water 
port (wide aperture) in the absence of ICMS. Similar to experiment 1, 
the difference between the S1 neuronal ensemble activity, recorded 
while the encoder rat examined the aperture with its whiskers in each 
trial, and an average template obtained previously, was utilized to 
create ICMS patterns applied to the decoder rat’s $1. We named these 
ICMS patterns virtual narrow and virtual wide. A total of 120 units 
and 223 multiunits were recorded in experiment 2. 

The BTBI accuracy for tactile information transfer was similar to 
that observed in experiment 1 (Figure 5 A-B). While encoder rats 
performed at 96.06 + 1.14% correct, decoder animals performed 
somewhat worse but significantly above chance (Percent correct: 
62.34 + 0.59%, range 60 - 64.58%; Binomial test: P < 0.05 in all 
sessions) (Figure 5 A-B and Figure 6 A-D). In this second experi- 
ment, the response latency of encoder rats was 2.66 + 0.1 seconds, 
while in decoders the latency was 2.68 + 0.09 seconds. 

To further demonstrate that the accuracy of the decoder rats’ 
performance was based on the ICMS patterns, which in turn were 
triggered by larger number of spikes produced by S1 neuronal 
ensembles, we compared the fraction of Virtual Narrow choices with 
the number of ICMS pulses delivered to the decoder’s S1 cortex. 
Increases in the number of ICMS pulses delivered to the decoder’s 
S1 were associated with a higher fraction of Virtual Narrow choices 
(= 25 pulses: 0.3966 + 0.04476 correct; >25 pulses: 0.5433 
0.02991 correct; Paired samples t-test = 2.321, df = 16, P 
0.0338) (see Figure 5 C and Figure 6 A-D ). Since ICMS patterns 
were directly derived, through a transfer function, from the neural 
ensemble activity recorded from the encoder animal’s S1 cortex in 
each trial, this result demonstrates that the decoder rat’s correct 
choices relied on the accuracy of the ICMS pattern in reproducing 
the number of action potentials generated by the real tactile stimulus 
information presented to the encoder rat. Feedback information, 
providing an additional reward to the encoder rat every time the 
decoder rat performed a trial correctly, also induced changes in the 
neural activity of the encoder rat. The encoder’s latency of response 
was similar after correct and incorrect trials (After correct = 2.6 
0.1 secs; After incorrect = 2.7 + 0.2 secs; Mann Whitney U 
19790, P = 0.49). However, similarly to the effects observed in 
experiment 1, the signal to noise ratio of neural activity in S1 also 
increased after an incorrect trial (Chi Square = 4.2, df=1; P = 
0.0404). 

It could be argued that the results reported here could have been 
obtained if prerecorded signals from encoder rats had been used to 
guide the behavior of the decoder rats. Qualitative and quantitative 
observation of the behavior of the animals reveals that this is not at all 
the case. In both motor and tactile BIBI sessions we observed drastic 
changes in the behavior of encoder and decoder rats as soon as they 
started to work as part of a dyad. Both encoder and decoder animals 
either made quick attempts to respond earlier or, conversely, they 
reduced their response rate or even stopped performing according to 
the dyad behavior. Thus, response latencies during motor BTBI ses- 
sions were largely increased for encoder animals (encoder training: 
14.77 + 0.9 seconds; encoder BTBI session: 20.06 + 1.0 seconds; t = 
3.975, df = 1170, P < 0.0001) and decreased in decoder rats (decoder 
training: 16.29 + 0.6 seconds; decoder BTBI sessions: 13.59 + 
0.5 seconds; t = 3.559, df = 1636, P = 0.0004). During the tactile 
BTBI sessions the responses latency was reduced in both encoder 
(encoder training: 5.40 + 0.6 seconds; encoder BTBI sessions: 2.66 + 
0.1 seconds; Mann-Whitney U = 13960, P < 0.0001) and decoder 
animals (decoder training: 4.632 + 0.6 seconds; decoder BTBI ses- 
sions: 2.68 + 0.09 seconds; t = 4.638, df = 12, P = 0.0006) as they 
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Figure 3 | Trial examples of a BTBI for transferring cortical motor signals. A) Examples of M1 neurons recorded while the encoder rat performed 

the task. Time = 0 corresponds to the lever press. Very different patterns of increased and decreased activity were observed before and after the lever press, 
suggesting that multiple task parameters were encoded by this M1 ensemble. B) Sample of trial by trial choices of the rat dyad (encoder and decoder) 
during execution of the motor task. The encoder’s performance is depicted by a blue line, while a red line indicates the decoder’s choices in the same trials. 
In trials 4,7,11 and 13 the behavioral response of the decoder rat did not match the one of the encoder. The overall performance of the decoder rat in this 
session was 69% correct. C) The bars represent the number of encoder’s M1 neuronal spikes recorded during each trial. The neuronal ensemble used in 
this session encoded very accurately each of the behavioral responses. D) Number of ICMS pulses delivered to the decoder’s M1 that resulted from the 


comparison of each trial in C to the template. 


started to work as a dyad. Therefore, the dyad performance depended 
on the nature of the task performed jointly by the animal pair. Likely 
the increased latencies observed in the motor task reflect the fact that 
pressing a lever is a learned artificial behavior, while the exploratory 
nose poking necessary for the tactile task is part of the rats’ natural 
behavioral repertoire. These overall changes in the dyad behavior, 
irrespective of their direction (e.g. increased or decreased latency), 
are a clear indicator that a fundamentally more complex system 
emerged from the operation of the BIBI; one which required con- 
siderable adaptation from the participant animals so that they could 
jointly perform the sensorimotor tasks. 

As the ICMS cues were delivered to primary cortical areas that are 
commonly involved in processing motor and somatosensory 
information in intact animals, we further asked how the decoder rat’s 
S1 cortex represented both real tactile stimuli, generated by mech- 
anical stimulation of its own facial whiskers, and ICMS signals repre- 
senting the encoder rat’s whisker stimulation, during operation of a 
BTBI. To measure this, we tested pairs of encoder and decoder rats 
during passive transmission of tactile information via a BTBI, while 


the whiskers of the encoder and decoder rats were mechanically 
stimulated. This experiment consisted of two parts: first, the encoder 
animal was lightly anesthetized and head fixed to an automated 
whisker stimulator that accurately reproduces the movement and 
speed at which the whiskers interact with the bars in the active tactile 
discrimination task (see Methods). The animal’s S1 neural activity 
following each movement of the bars was analyzed in real time and 
delivered, as an ICMS pattern, to the decoder rat’s $1. Meanwhile, the 
decoder rat remained in an open field in a different room while its S1 
neural activity was recorded. After this phase was completed, the 
decoder animal was also lightly anesthetized and placed in the auto- 
mated whisker stimulator. This allowed us to determine how the 
decoder rat’s S1 neuronal sample, that responded via the BTBI to 
the tactile stimuli delivered to the encoder’s whiskers, responded to 
tactile stimuli elicited by passive whisker stimulation of their own 
vibrissae. 

Passive whisker stimulation, in either the encoder or decoder rats, 
induced significant firing modulations in the decoder rat’s $1. These 
were characterized by clear increases of firing activity occurring 
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Figure 4 | Experimental apparatus scheme of a BTBI for transferring cortical tactile information. A) In the tactile discrimination task, the encoder 
animal was required to sample a variable width aperture using its facial whiskers. The width could be “Narrow” as shown in the left photograph, or 
“Wide”. After sampling, the encoder animal had to report whether the aperture was narrow or wide by nose poking on a left or right reward port 
respectively. If correct, the animal received a small water reward. As the encoder explored the aperture, a sample of its S1 activity was recorded, compared 
with a template trial and then transferred to the decoders’ S1 via ICMS. The pattern of microstimulation constantly varied according to the number of 
spikes recorded from the encoder rat’s S1 in each trial. The decoder rat was required to make a response in the reward port corresponding to the width 
sampled by the encoder, guided only by the microstimulation pattern. If the decoder rat accurately responded in the correct reward port, both rats 
received a small water reward. Thus, the encoder rat received an additional reward in case both animals of the dyad performed a trial successfully. 


immediately after the moving bars touched the whiskers of each 
animal (see Figure 7 A and B). These significant $1 neuronal res- 
ponses occurred in 70.91% (39/55 multiunits) of the microwires 
implanted in the decoder rat’s $1, which were used to deliver 
ICMS patterns through the BTBI, and in 93.06% (67/72 multiunits) 
of the microwires from which S1 neuronal activity was recorded from 
decoder rats (see Figure 7 B and C). The magnitude of the S1 tactile 
responses elicited by mechanical stimulation of the decoder rat’s 
facial whiskers was 4.82 + 0.4 spikes/trial and the duration was 
111.4 + 11 ms. During the BTBI transmission, ICMS of the deco- 
der’s S1 induced a significant increase of S1 neurons firing activity 
lasting for 119.7 + 20 ms. Due to the microstimulation artifact in the 
recordings, we focused our analysis on the firing activity increases 
occurring after the last pulse of microstimulation was transmitted 
(see red traces Figure 7 C). 

Analysis of the data obtained during passive BTBI communication 
further demonstrated that S1 neurons in the decoder’s brain 
responded differently for each of the virtual tactile stimuli. More 
than half of the S1 multiunits recorded presented differential firing 
rates for Virtual Wide and Virtual Narrow stimuli (28/44 = 63.64% 
multiunits). Also, the Virtual Narrow stimulus was characterized by 
higher neuronal response magnitudes (Virtual Narrow: 3.861 + 
0.6229 spikes/trial; Virtual Wide: 2.200 + 1.079 spikes/trial; 
Wilcoxon sum of ranks = 197; P = 0.0182) and durations (Virtual 
Narrow: 102.7 + 16.28 ms; Virtual Wide: 31.54 + 14.85 ms; 
Wilcoxon sum of ranks = 200; P = 0.0074). To measure whether 
the differences in firing rates were due to discrimination or due to an 
‘upstate’ related to the repeated microstimulation, we also compared 
which S1 multiunits exhibited different firing rates for Virtual 
Narrow and Virtual Wide. From the total of S1 multiunits that dis- 
played differences in firing rates for the discrimination period, we 
found that 35.7% (10/28 multiunits) had no significant differences in 
the baseline firing rate. Thus, more than one third of the $1 multi- 
units recorded showed no signs of an ‘upstate’ in their baseline due to 
repeated microstimulation. This supports the hypothesis that after 


the decoder rats learned to use the BTBI, via ICMS cues, their S1 
became capable of accurately representing, processing, storing and 
recalling information about both the tactile stimuli delivered to its 
own whiskers, as well as mechanical displacements of the encoders’ 
facial vibrissae. 

Finally, to further demonstrate the range of potential operation of 
our BTBI preparation, we tested whether a long-distance commun- 
ication of a rat dyad, with the encoder rat performing the tactile 
discrimination task at the ELS-IINN (Natal, Brazil) and the decoder 
rat receiving patterns of microstimulation and responding at Duke 
University (Durham, USA), would be capable of performing the 
same task. For this, neural activity recorded from S1 of the encoder 
rat performing the tactile discrimination task was sent via an internet 
connection and delivered, as an ICMS pattern, to the decoder rat S1 
(Figure 8). Even under these extreme conditions, the BIBI was also 
able to transfer in real-time behaviorally meaningful neuronal 
information. Although the mean time of data transmission observed 
in this long-distance BTBI was increased from 20 ms (during trans- 
mission in our Duke lab) to 232 + 217.5 ms, a similar number of 
correct responses was found (short distance transmission: 62.34 + 
0.59%; long distance transmission: 62.25% + 0.71) in 26.5 + 
0.5 trials in the decoder animals. 


Discussion 

The present study demonstrates for the first time that tactile and 
motor information, extracted in real time from simultaneously 
recorded populations of cortical neurons from a rat’s brain, can be 
transmitted directly into another subject’s cortex through the util- 
ization of a real-time BTBI. Operation of a BTBI by an encoder- 
decoder rat dyad allowed decoders to rely exclusively on neural pat- 
terns donated by encoders in order to reproduce the encoder’s beha- 
vioral choice. ICMS patterns reflecting the number of action 
potentials recorded from either the encoder rat’s M1 or S1 during 
a single trial were sufficient for decoder rats to repeatedly perform 
two different tasks, significantly above chance levels, in real-time. 
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Figure 5 | Behavioral performance using a brain-to-brain interface to 
transfer cortical tactile information. A) Performance of encoder and 
decoder animals during operation of a BTBI for tactile information 
sharing. Notice that the performance of the encoder animals was above 
85% in all sessions. The performance of the decoder animals was above 
60% in all sessions presented and immediately dropped to chance levels 
when the cable was disconnected but the system remained fully functional. 
B) Performance of all decoder animals analyzed with a moving average of 
10 trials. C) The panel depicts the fraction of the decoder’s responses in the 
Narrow reward port after different patterns of microstimulation were 
delivered. As the number of microstimulation pulses increased a higher 
fraction of responses was observed in the Narrow reward port (Virtual 
Narrow choice), suggesting that the microstimulation threshold of 
response for decoder animals was situated between 26-40 pulses. 


Interestingly, half of the number of pulses used to stimulate the 
decoder’s M1 were sufficient to successfully deliver a message to 
the decoder’s S1 cortex, suggesting that primary sensory cortical 
areas may have a lower threshold to operate a BTBI. 


We also demonstrated that operation of a BIBI induced differ- 
ential patterns of activation in the decoder rat’s $1. Thus, the same S1 
neurons that responded to the mechanical stimulation of the decoder 
rat’s own whiskers were capable of representing information derived 
from stimulation of the encoder rat’s whiskers via the BTBI. 

Additionally, continuous operation of the BTBI also affected the 
behavior and neural activity of the encoder rat, which was able to 
reduce its response latency and increase the signal/noise ratio of its 
S1/M1 neuronal activity in response to an error by the decoder rat. As 
far as we can tell, these findings demonstrate for the first time that a 
direct channel for behavioral information exchange can be estab- 
lished between two animal’s brains without the use of the animal’s 
regular forms of communication. Essentially, our results indicate 
that animal brain dyads or even brain networks could allow animal 
groups to synchronize their behaviors following neuronal-based 
cues. 

Successful BTBI operation required four simultaneous conditions 
to be present: first, the encoder animals had to achieve a very high 
level of performance in both tasks. As proof, only one successful 
BIBI session was obtained when the encoder rat’s performance 
was below 80%. Second, recordings from the encoder’s cortex had 
to yield stable neural ensemble activity which was highly correlated 
to the behavior that needs to be encoded by the BTBI. Note that 
successful BTBI operation was achieved using information collected 
from random ensembles of neurons dispersed within each cortical 
area. This finding indicates that information was not anatomically 
segregated either in S1 or in M1. Third, the midpoint of the sigmoid 
transfer function (which was set at the beginning of the session) had 
to closely match the midpoint of the neural function that represented 
the two stimuli/actions. We found that such a midpoint tended to be 
the same for each cortical region, suggesting that groups of neurons 
with similar physiological profiles were recorded in most cases. 
Fourth, our results showed that both encoder and decoder rats chan- 
ged their behavior according to the dyad performance. This obser- 
vation suggests that operation of a BTBI induces the establishment of 
a highly complex system, formed by a pair of interconnected brains. 
As such, this brain dyad behaved in a way that could not be predicted 
if only pre-recorded neural signals had been used for encoding pur- 
poses. We speculate that the description of the complex system gen- 
erated by the dyad transferring information and collaborating in real 
time, will reveal fundamental properties about the neural basis of 
communication and social interactions’*”’. 

Although we have shown accurate transfer of brain-derived motor 
and sensory information through a BTBI, it remains to be explained 
how the brain simultaneously integrates information generated by 
direct ICMS and by natural stimuli (e.g. real whisker stimulation). 
Previous studies in rhesus monkeys have shown that the brain is able 
to decode highly complex ICMS patterns in a single trial’. 
Specifically, it has been shown that a brain-machine-brain control 
loop allows for continuous update of information in the S1 cortex, 
while a monkey explores a virtual tactile stimulus. The effects of a 
neuroprosthetic’s operation on cortical neuronal responses have also 
been studied in the representation of the rat forelimb sensorimotor 
cortex, where it was shown that information flow can be altered by S1 
microstimulation””’. Lastly, a recent study has shown that the abil- 
ity to use a BMI is mediated by the striatum in mice*. Altogether, this 
body of evidence supports the notion that continuous use of ICMS to 
deliver information to the brain is associated with plastic changes in 
neuronal ensemble responses in cortical and subcortical regions. The 
data obtained here during passive BTBI operation supports this con- 
clusion by showing that as animals learned to use the microstimula- 
tion cues, differential patterns of $1 neuronal responses emerged for 
each of the virtual tactile stimuli. This finding is consistent with our 
previous observation that S1 neurons undergo significant functional 
plasticity during the period in which rats learn a tactile discrimina- 
tion task’. Accordingly, our results further suggest that successful 
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Figure 6 | Trial examples of a BTBI for transferring cortical tactile signals. A) Examples of $1 neurons recorded while an encoder rat performed 
the aperture discrimination task. Time = 0 corresponds to the moment the animal breaks the photo beam in front of the discrimination bars. B) Blue lines 
represent the choices of the encoder rat and red line represents the choices of the decoder rat. In trials 8, 15 and 17 the decoder rat selected the incorrect 
reward port. C) Number of action potentials recorded from 3 S1 neurons in each trial after the whiskers sampled the discriminanda. Typically, a higher 
spike count was found for narrow trials, when compared to wide trials. D) Number of pulses delivered to the $1 cortex of the decoder rat in each trial. The 
number of pulses delivered to the S1 cortex of the decoder rat was directly derived from the number of spikes present in the encoder animal in each trial. 
The overall performance achieved by the rat dyad in this session was 64% correct trials. 


BTBI operation is fundamentally linked to the ability of S1 ensembles 
to undergo plastic reorganization in response to microstimulation 
patterns”. 

Altogether, the results described here indicate that the channel 
capacity (amount and precision of information, bandwidth) and 
the dynamic properties of cortical neuronal ensembles are the two 
major determinants of the amount and quality of information that 
can be transferred between animal brains via a BIBI. Thus, beyond 
the neurobiological challenge of understanding how the brain inte- 
grates natural and virtual stimuli, a second class of problems directly 
related to the characteristics of the BTBI as a channel for information 
transfer must be addressed. In general terms, the BTBI can be 
described as a discrete noisy channel, meaning “a system whereby 


can be transmitted from one point to another ”**. The limit for the 
amount of information that can be transferred by unit of time (i.e. 
capacity)” is currently unknown for a BTBI channel. In the tasks 
used here, the minimum and maximum inputs depended on the 
range of the firing rate in the neurons used, while the output 
depended mostly on the electrical microstimulation threshold of 
cortical ensembles in the decoder’s brain. However, channel capacity 


can be increased while still using the rationale described in Figure 1 
(neural data - transfer function - stimulation delivery). For example, 
it will be important to test in the future the effect of other types of 
inputs (e.g. larger neuronal ensembles; Local Field Potentials), trans- 
fer functions (e.g. exponential, linear, step functions) and outputs 
(e.g. one versus several pairs of microelectrodes used for ICMS, dis- 
posed in 2D or 3D cortical space, delivering photostimulation instead 
of electrical current) on the overall dyad performance. In this con- 
text, we expect that the use of newly introduced microelectrode 
cubes, created in our laboratory, that spread across 3D cortical space, 
to deliver spatiotemporal patterns of information from the encoder’s 
brain to the decoder’s will provide a significant increase in BTBI 
bandwidth, likely leading to a substantial improvement in the overall 
animal dyad performance. 

Lastly, it is important to stress that the topology of BTBI does not 
need to be restricted to one encoder and one decoder subjects. 
Instead, we have already proposed that, in theory, channel accuracy 
can be increased if instead of a dyad a whole grid of multiple recip- 
rocally interconnected brains are employed. Such a computing struc- 
ture could define the first example of an organic computer capable of 
solving heuristic problems that would be deemed non-computable 
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Figure 7 | Neural activity in the decoder brain discriminates stimuli applied to the encoder’s whiskers. PSTHs on the left panels show S1 neuronal 
responses during the wide tactile stimulus whereas PSTHs on the right panels depict narrow tactile stimulus. The top and middle panels show S1 activity 
recorded in anesthetized encoder and decoder rats while their facial whiskers were passively stimulated by a set of moving bars. The moving bars generate a 
tactile stimulus exactly like the one produced during the tactile discrimination task. The lower panels represent the decoder rat’s S1 activity while receiving 
ICMS (red traces) via a BTBI that transmitted tactile information from an anesthetized encoder rat which was having its whiskers passively stimulated. 
Time zero in all panels corresponds either to the tactile stimulus or the last microstimulation pulse. A) A clear peak of S1 activity can be observed 
immediately after the encoder’s whiskers contacted the bars (other peaks occurred due to rebounding of the moving bars). Increased counts of action 
potentials were typically associated with the narrow stimulus (compare peaks in left versus right panels). B) Like encoder rats, when the decoder rats’ 
whiskers were passively stimulated by the moving bars, clear peaks of S1 activity with different heights can be observed (see left versus right panels). 
C) When the encoder rats’ whiskers were passively stimulated (shown in A) and the BTBI was used to transfer tactile information in real time (shown in 
C), clear increases in activity were observed in the decoder’s S1 cortex after time 0. These S1 firing modulations were larger when the narrow stimulus was 
applied to the encoders’ whiskers when compared to the wide stimulus (see left versus right panels) and were observed in the same S1 neuronal ensembles 
that responded to natural whisker stimuli (shown in B). Thus, the S1 neuronal responses observed in the decoder rat demonstrate that it learned to use the 
BTBI and that a representation of the tactile stimuli applied to the encoders’ whiskers could be superimposed on the preexisting representation depicting 
tactile stimuli applied to its own facial whiskers. 
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Figure 8 | Intercontinental brain-to-brain interface to transfer cortical 
tactile information. To test the full potentialities of the BTBI, a 
brain-to-brain interface to transfer cortical tactile information was 
established between our laboratory at the IINN-ELS in Brazil and our 
laboratory at Duke University in the USA. An encoder rat performed a 
tactile discrimination task at the IINN-ELS. Meanwhile its neuronal 
activity in S1 was recorded and sent over the internet to our laboratory at 
Duke University. The sigmoid transformation algorithm was used to 
transfer the number of action potentials into microstimulation patterns 
that there were then delivered to the decoder rat’s S1 cortex. As the decoder 
rat made a behavioral response, feedback was sent over the internet to the 
encoders’ chamber back at the INN-ELS. 


So 


by a general Turing-machine. Future works will elucidate in detail 
the characteristics of this multi-brain system, its computational cap- 
abilities, and how it compares to other non-Turing computational 
architectures”. 


Methods 


All animal procedures were performed in accordance with the National Research 
Council’s Guide for the Care and Use of Laboratory Animals and were approved by 
the Duke University Institutional Animal Care and Use Committee. Long Evans rats 
weighing between 250-350 g were used in all experiments. 


Motor brain-to-brain interface. The behavioral motor task consisted of a dark 
operant chamber equipped with two levers, one LED (Light Emitting Diode) above 
each lever and, on the opposite wall, a water reward port. Animals were trained to 
press one of two levers, cued by an LED turned on at the beginning of each trial. A 
correct choice opened the reward port and allowed brief access to water (300 ms). 
When animals reached stable performances above 80% correct choices they were 
assigned either to an encoder or decoder group. The operant chamber configuration 
remained similar in both the encoder and the decoder groups. Animals assigned to the 
encoder group were implanted with recording arrays of 32 microelectrodes in the 
primary motor cortex and after recovery resumed the initial training scheme. 
Animals assigned to the decoder group were implanted with arrays of 4 to 6 
microstimulation electrodes in the primary motor cortex and were further trained to 
associate the presence of electrical microstimulation pulses with the correct lever 
press. Extra training followed, with a sequence of 60 to 100 pulses indicating a correct 
choice in the right lever while the absence of microstimulation pulses (1 pulse) 
indicated a correct left lever choice. During the electrical microstimulation training 
phase a trial started with a brief period of white noise, followed by the electrical 
microstimulation cue. Immediately after this cue both LEDs were turned on. If a 
correct choice was made the reward port would open and the animal was allowed a 
brief period of access to water (300 ms), otherwise both LEDs were turned off and the 
intertrial interval started. 

When decoder animals reached stable performances above chance, brain-to-brain 
interface (BTBI) sessions with motor activity were performed. These sessions were 
composed of three different phases: 1) identification of cells accurately encoding the 
motor action performed, 2) data collection for template trials during a rat-to-brain 
mode and, 3) brain-to-brain interface using real-time analysis followed by electrical 
microstimulation. There were no setup differences between the rat-to-brain mode 
and the brain-to-brain interface. The only difference between these two conditions 


was that in the rat-to-brain mode the patterns of microstimulation depended on the 
behavior of the encoder animal, while during the brain-to-brain interface mode the 
patterns of microstimulation depended solely on the neural activity of the encoder rat. 
Neural activity was first studied in encoder animals performing the behavioral task 

to identify units that accurately encoded for the motor activity associated with each 
lever press. During the rat-to-brain mode the operant chambers remained as in the 
training phase, however the presence of a correct choice in the encoder operant 
chamber activated the pattern of electrical microstimulation cue corresponding to the 
same lever on the decoder animal operant chamber. A correct choice by the decoder 
rat was signaled by a tone and both encoder and decoder animals were allowed a brief 
period of access to water. The goal of this phase was to create a template trial based on 
the neural activity of the encoder rat pressing one of the levers in several trials. 

During the brain-to-brain interface mode the number of action potentials in each 
rial was compared to the template trial and the Zscore of the difference between them 
was used to determine the number of pulses present in the pattern of microstimu- 
ation. A sigmoid function was used to transfer the Zscore value to the number of 
pulses present in the microstimulation pattern. A higher Zscore was associated with 
an increased number of pulses in the microstimulation cue (i.e. right lever) while a 
ower Zscore was associated with a decreased number of pulses in the microstimu- 
ation cue (e.g. left lever). 


Active tactile brain-to-brain interface. Mildly water deprived animals were trained 
o perform a behavioral discrimination task as previously described””. Briefly, this task 
required animals to discriminate between a wide or narrow aperture in order to 
receive a water reward. The animal was placed in the behavioral box compartment 
where it waited for the central door to open and allow access to the second 
compartment, the inner chamber. After the animal entered the inner chamber, it had 
to pass through the variable width discrimination bars and then nose poke the center 
of the front wall. The nose poke in the inner chamber opened two water reward pokes 
located in the outer chamber from which the animal had to select one. The reward 
poke on the right corresponded to the wide aperture, whereas the poke on the left 
corresponded to the narrow aperture. As the animal chose from one of the reward 
pokes, the door separating the inner and outer chambers closed. Correct responses 
were rewarded by 50 il water rewards. Incorrect responses were followed by 
immediate closing of the reward pokes. The percent of trials performed correctly was 
used as a measure of tactile discrimination. 

Animals were then evenly assigned to encoder or decoder groups. Encoder animals 
(n = 2) were implanted with recording arrays of 32 microelectrodes in the right S1 
and after recovery resumed the initial training scheme. Animals assigned to the 
decoder group were implanted with arrays of six microstimulation electrodes in the 
right S1 (n = 7). In six decoder animals, recording electrode arrays were also 
implanted either in the right (N = 1) or in the left S1 (N = 5). After recovering from 
surgery decoder animals were further trained to associate the presence of electrical 
microstimulation pulses with the correct lever press. Extensive training followed, with 
a sequence of 50 pulses indicating a correct choice in the left reward poke while the 
absence of microstimulation pulses (1 pulse) indicated a correct choice in the right 
reward poke. Decoder animals were required to identify the microstimulation cue and 
associate it with a behavioral response in one of the reward pokes. A brief tone 
indicated the beginning of the trial immediately followed by the microstimulation 
cue. After a period of 500 ms both reward pokes would open and the rat was required 
to make a response in one of the photo beams. A correct choice was followed by a brief 
tone and access to water. When decoder animals reached stable performances of 
> 65% correct trials for 3 consecutive sessions, tactile BTBI sessions began. 

Neural activity was first studied in encoder animals performing the behavioral task 
to identify units that accurately encoded for the tactile stimuli associated with sam- 
pling the width between bars. During the rat-to-brain mode the operant chambers 
remained as in the training phase, however the microstimulation cue presented to the 
decoder animal always matched the stimulus presented to the encoder animal. After a 
correct response by the encoder rat, a brief tone followed by a microstimulation cue of 
1 or 50 pulses was sent to the decoder animal and both reward ports in the second 
chamber would open. If the decoder rat accurately discriminated the microstimula- 
tion cue both rats were rewarded. During the brain-to-brain interface mode the 
neural activity of the encoder rat was analyzed from the moment that the rat broke the 
discrimination bars photo beam to the moment that the rat broke the photo beam in 
the center poke. The number of action potentials found in this interval was then 
counted and compared to the distribution of the Zscores relative to the spikes present 
in all the previous Wide trials. A Zscore was determined and transferred using a 
sigmoid function, into the number of pulses present in the pattern of microstimu- 
lation. 


Passive tactile brain-to-brain interface. The encoder animal was anesthetized and 
remained head fixed in one room, while the decoder rat was in an open field in a 
different location with the neural activity also being recorded. The encoder rats’ 
whiskers were then stimulated by a set of moving bars that accurately reproduce the 
dynamics observed in the active tactile width discrimination task"*. The bars were set 
up for Wide or Narrow widths and, for each width, the head fixed animal was 
stimulated for approximately 6 minutes at 0.3 Hz. Neural activity was first analyzed 
in real time and units with clear whisker related activity were used for the session. To 
establish a baseline distribution; an initial group of 100 wide stimulus trials was 
recorded. Then the passive brain-to-brain interface mode started. The encoders’ 
whiskers were stimulated by the moving aperture corresponding to a Wide stimulus. 
Meanwhile the number of action potentials recorded from the encoder animal was 
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counted. The number of action potentials was compared to the distribution of action 
potentials found at the baseline at the beginning of the session and a Zscore was 
calculated. This Zscore was transferred into the number of pulses to be used in the 
microstimulation using a sigmoid function. The decoder rat then received the pattern 
of microstimulation derived from the sigmoid function. Immediately after the 
encoder animal had been passively stimulated with both Wide and Narrow widths, 
the decoder animal was anesthetized, head fixed, and its whiskers were passively 
stimulated with the same Wide and Narrow stimuli as the encoder animal. 


Surgery for microelectrode array implantation. Fixed or movable microelectrode 
bundles or arrays of electrodes were implanted in the M1 and S1 of rats. Craniotomies 
were made and arrays lowered at the following stereotaxic coordinates for each area: 
S1 [(AP) —3.0 mm, (ML), +5.5 mm (DV) —0.7 mm], M1 [(AP) +2.0 mm, 

(ML) +2.0 mm, (DV) —1.5 mm]. 


Electrophysiological recordings. A Multineuronal Acquisition Processor (64 
channels, Plexon Inc, Dallas, TX) was used to record neuronal spikes, as previously 
described’’. Briefly, differentiated neural signals were amplified (20000-32,000 x) 
and digitized at 40 kHz. Up to four single neurons per recording channel were sorted 
online (Sort client 2002, Plexon inc, Dallas, TX ). Online sorting was validated offline 
using Offline Sorter 2.8.8 (Plexon Inc, Dallas, TX). 


Intracortical electrical microstimulation. Intracortical electrical microstimulation 
cues were generated by an electrical microstimulator (Master 8, AMPI, Jerusalem, 
Israel) controlled by custom Matlab script (Nattick, USA) receiving information from 
a Plexon system over the internet. Patterns of 1-100 (bipolar, biphasic, charge 
balanced; 200 j1sec) pulses at 400 Hz (motor BTBI) or 250 Hz (tactile BTBI) were 
delivered to the cortical structures of interest (M1 and S1 respectively). Current 
intensity varied from 38-200 1A (motor BTBI) and 30-240 tA (tactile BIBI). 


Data analysis. For both behavioral tasks the number of correct responses was used as 
a measure of behavioral performance. We also analyzed the animals’ response latency 
as a measure of independency between the performance of each animal alone or in a 
dyad. 

Neuronal data were processed and analyzed using Neuroexplorer (version 3.266, 
NEX Technologies) and custom scripts written in Matlab (7.9.0, Mathworks, Natick, 
MA). Statistical significance of neural responses was evaluated using a method based 
on cumulative-summed spike counts*”’. Comparisons of characteristics of neural 
responses for different conditions were performed using non-parametric tests 
(Mann-Whitney- Wilcoxon or Kruskal-Wallis). Signal-to-noise ratio of neural res- 
ponses was calculated as the proportion of responses, occurring after a correct or 
incorrect decoder response, that presented Zscore absolute values above 0.3 standard 
deviations. This specific value was used because it corresponded to the midpoint of 
the sigmoid curve). Statistical significance was determined using a chi square test for 
proportions. 
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Published: 9 July 2015 : the movements of artificial devices. Here, we introduce a Brainet that utilizes very-large-scale brain 

activity (VLSBA) from two (B2) or three (B3) nonhuman primates to engage in a common motor 
behaviour. A B2 generated 2D movements of an avatar arm where each monkey contributed equally 

: to X and Y coordinates; or one monkey fully controlled the X-coordinate and the other controlled the 

: Y-coordinate. A B3 produced arm movements in 3D space, while each monkey generated movements 
in 2D subspaces (X-Y, Y-Z, or X-Z). With long-term training we observed increased coordination of 

: behavior, increased correlations in neuronal activity between different brains, and modifications 

: to neuronal representation of the motor plan. Overall, performance of the Brainet improved owing 

: to collective monkey behaviour. These results suggest that primate brains can be integrated into a 

: Brainet, which self-adapts to achieve a common motor goal. 


BMIs are computational systems that link biological circuits to external devices, such as computer cur- 
sors, robotic prostheses and communication aids'. Heretofore, BMIs have been utilized either: (i) to 
extract motor signals from neural activity and convert them to the movements of external devices””, (ii) 
to deliver sensory signals from the environment to the brain®, or (iii) to combine both operations and 
enable bidirectional communications between the brain and machine’. In each of these implementations, 
a BMI serves as an accessory to a single brain. Recently an entirely new direction was proposed for BMI 
research — a brain to brain interface (BtBI)!°. BtBI allows animal brains to exchange and share sensory 
and motor information to achieve a behavioural goal'!!*"!”. BtBI is a hybrid computational system since 
it incorporates both biological components (the primate brains) and digital parts (the BMI system). 
: In the present study, we have designed and tested a more elaborate computational architecture which 
: we refer to as a Brainet’®. Our Brainets involved groups formed by 2-3 monkeys in a shared BMI that 
: enacted conjoint motor behaviours. Previously, human psychophysics studies have shown that two or 
more individuals who are performing movements simultaneously often entrain to each other's behavior, 
even if they are not explicitly instructed to do so’*”. However, the neurophysiological mechanisms of 
such joint actions are not well understood. In particular, we were interested in investigating the possibil- 
ity that neuronal ensemble could directly control conjoint behaviors enabled by multiple interconnected 
BMIs. 
Our study adds to previous attempts to overcome limitations of one individual confronted with a high 
: processing load by mixing contributions of multiple individuals****. Particularly relevant to our present 
: work, several EEG studies'**°** have combined brain derived signals from multiple subjects to enhance 
: visual discrimination, motor performance, and decision making. A recent EEG study*° has implemented 
: shared control that involved dynamic collaboration of multiple individuals in real time to achieve a 
common goal. However, in none of these EEG experiments participants interacted with each other over a 
long term. Moreover, no large-scale intracranial cortical recordings were obtained in order to investigate 
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Figure 1. Experimental setups for B2 and B3 experiments. (A) Monkeys were seated in separate 

rooms, each facing a computer monitor showing the virtual avatar arm (inset in C) from a 1st person 
perspective. (B) Shows the shared control task, (X,Y) position of the virtual arm was decoded during 
centre-out movements from the two monkeys’ brains with each given 50% control of the arm. Electrode 
array location shown on brains. (C) Shows the partitioned control task. X position of the arm was decoded 
from one monkey and Y position from the other during centre-out movements toward targets. (D) Shows 
the 3-monkey task. Each monkey observed and had 50% control over 2 of the 3 dimensions (X, Y, or 

Z). Together, the three monkeys must accurately perform a 3-D centre-out movement to achieve reward. 
Drawings by Miguel A.L. Nicolelis. 


the neurophysiological mechanism underlying conjoint motor behaviour. As such, the present study is 
the first to implement Brainets based on the chronic extracellular recordings of up to 783 individual 
cortical neurons distributed across multiple rhesus monkey brains. 

To this end, we have implemented Brainet architectures consisting of two (B2) (Fig. 1B,C) or three 
(B3) (Fig. 1A,D) monkey brains. These Brainets (i) controlled 2D/3D movements of an avatar arm by 
sharing signals derived from multiple brains (Fig. 1B,D), (ii) partitioned control by delegating subtasks 
to different subjects (Fig. 1C,D) and (iii) enabled a super-task that was composed of individual BMI 
tasks (Fig. 1A,D). 


Results 

Four monkeys participated in the experiments. They were chronically implanted with multielectrode 
arrays in motor (M1) and somatosensory cortices (S1)!*°. Extracellular electrical activity from 570-783 
neurons was sampled chronically from monkeys M (410-501 neurons), C (196-214), O (156-229), and 
K (140-156) up to five years after implantation**. B2s operated using at least 550 neurons, while the 
B3s operated with at least 775 neurons. During experimental sessions, monkeys sat in separate rooms 
where each of them viewed a realistic monkey avatar arm on a computer screen (Fig. 1A). Movements 
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Group name | Acquisition time | Precision | Hit rate | 

HC Dyad MO 0.7s ] 2.04 82.02% 

B2: Dyad MO 1.86s 2.04 86% 

B2: Dyad MC 1.75s 2.04 80% 

B3: Triad MCK 3.31s 0.37 78% 
Table 1. 


of the avatar arm were generated using either a hand-held joystick (hand control) or from modula- 
tions in VLSBA in a mode called brain control*». In brain control, a neural decoding algorithm called 
the unscented Kalman filter (UKF)*” extracted reach kinematics from neuronal ensemble activity in 
real-time. 


Experiment 1: Shared control. In the first experiment (shared control task, Fig. 1B), two monkeys 
worked together to move the avatar arm in 2D space. Each monkey contributed equally (50%) to both 
the X and Y coordinates. The monkeys moved the avatar arm from the screen centre towards a circular 
target (Movies S1, $2). These movements were enacted either using the joystick (dyad M&O) or the 
Brainet (monkey dyads M&O and M&C). To obtain training data for brain control, we started each 
session with a passive observation epoch during which monkeys watched the computer-generated avatar 
arm movements along centre-out trajectories**. One UKF was fit per monkey, and the UKF outputs (X 
and Y coordinates) were combined across monkeys to produce the conjoint brain-controlled movements 
of the avatar arm. 

The behaviour of the dyad MO improved significantly with training (Fig. 2A-D). Over several weeks 
of training (4 weeks of hand control followed by 3 weeks of brain control), we observed a significant 
reduction in target acquisition time between early and late sessions (Figs. 2A,B) (p< 0.001 and p < 0.01; 
KS test for hand control and brain control, respectively), trial duration (p< 0.001 and p< 0.02) and 
inter-reward interval (p < 0.004 and p < 0.02). The mean trial duration across weeks of training signifi- 
cantly decreased for both hand control (At = 0.13, 1-way ANOVA: p < 0.05) (Fig. 2C) and brain control 
training (At=0.7s, 1-way ANOVA: p< 0.05) (Fig. 2D). To assess the performance of the B2 in manual 
control and brain control modes, we also calculated performance metrics reported in the previous lit- 
erature*”“°. Table 1 shows the overall mean target acquisition time, target hit rate and the movement 
precision required, which was quantified as the ratio of target size to the size of the workspace*. 

As monkeys M and O learned the shared control task, their coordination improved in both hand 
control and brain control modes (Fig. 2E-J). In the initial session, with monkey M initiating move- 
ments earlier than monkey O (Fig. 2E,I; hand control), the lag between the monkeys’ reaction times 
was 200+ 12ms (mean+SEM). With conjoint training, reaction time lag between the two monkeys 
steadily decreased to 10-27 ms over 21 sessions (13 sessions of hand control and 8 sessions of brain 
control, Fig. 21). This gradual reduction in lag was statistically significant (hand control: linear regression: 
r°=0.57; p<0.01, brain control: linear regression: r?>=0.53; P< 0.05). As the monkey dyad became 
more coordinated, the cortical neural activity lag between the two brains decreased and stabilized near 
zero (Fig. 2G,H,J). More specifically, cortical activity lag decreased during hand control (linear regres- 
sion: r7= 0.41; P< 0.05) and remained approximately zero (4.2 + 5.1 ms) throughout brain control (purple 
data in Fig. 2J, linear regression: R?=0.01; P > 0.05). 

For the other dyad (M&C), the behavioural responses were synchronous and lag remained close to 
zero (3+ 35 ms; linear regression: P = 0.44) throughout all sessions (8 sessions) (Fig. S1, Movie S2). Dyad 
M&C performed brain control after passive observations of the avatar arm movements and without a 
requirement to coordinate their behaviours in the manual task. This training sequence probably aided 
better coordination between the subjects. Table 1 shows the mean target acquisition time, target hit rate 
and the movement precision required for the dyad M&C. 

Concurrent changes in neural activity during Brainet operation could be related to multiple factors: 
common visual feedback of the avatar arm movements, common representation of the target (spatial loca- 
tion and temporal onset), and common BMI outputs (movements to the same location). To understand 
if the interplay between the common sensory input and common motor behaviour could account for 
the genesis of concurrent neural activity we carried out further analyses. First, we investigated the rela- 
tionship between the behavioural outcomes in each trial and concurrent neuronal modulations between 
the monkeys in the same trials. We found correct task outcomes were associated with greater neural 
correlations between monkeys, both in hand control (Fig. 3A; P < 0.01; unpaired t-test) and brain control 
(Fig. 3B; P< 0.01). To clarify the role of concurrent neuronal activity further, we conducted an analysis 
where a k-NN classifier predicted the trial outcome, based on the neural correlations between monkeys, 
both before (700 ms before target appearance until target appearance) and after target onset (from 100 
to 800 ms after target appearance). The neuronal correlations were significant predictors of trial outcome 
(P <0.05, 1-proportion z-test) not only after target appearance but also before (Fig. 3C,D). Notably, the 
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Figure 2. B2 shared control task. (A,B) Boxplots comparing target acquisition time (left panel), trial 
duration (centre panel), inter-reward interval (right panel) in hand control (A) and in brain control 

(B). (C,D) Changes in trial duration with conjoint training across weeks. Mean + SEM trial duration for 
each of the four weeks of hand control experiments (C) and three weeks of brain control (D) for dyad 
M&O. Dashed line shows linear regression fit. Trial duration reduced significantly with improvement in 
coordination between monkeys. (E,F) Trial-averaged movement profiles from an early session (dashed) and 
a late session (solid) for monkey M (red) and O (blue) aligned to the time of target onset (time = 0 ms.). 
Plots are shown up to 1.5s for hand control (with mean target acquisition time of 0.8s) and 3s for brain 
control (with mean target acquisition time of 2.2s). The target was located 9cm from the centre (y-axis). 
As the dyad trained together over a span of 7 weeks the reaction time lag, the time of movement onset 

of monkey O relative to monkey M, decreased during both hand control (E) and brain control (F) (G,H) 
Neural activity PETH from an early session and a late session for hand control (G) and brain control 

(H) Each row is a single neuron, colour denotes normalized firing amplitude. Monkey M and O neurons 
marked by black (M) and grey (O) vertical bands. Notice that neuronal modulations are more intense and 
synchronized across the monkey pair in the later session as compared to the early session. (I) Changes 

in reaction time lag between monkey M and O across experiments. Lag was derived from peak of cross- 
correlation of two monkeys’ behavioural traces (E,F) Trends fit with linear regression. As the dyad trained 
together the reaction time lag reduced to 10 + 27 ms. (J) Lag in neural activity between two monkeys over 
the same experiments as in (E) computed again by finding the peak of cross correlogram on each session. As 
the dyad trained together the reaction time lag reduced to 4.24 5ms. 
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Figure 3. (A-B) Cross correlogram of Monkey M vs. Monkey O neural activity on correct trials (upper 
left panel) and error trials (lower left panel) for both a representative hand control (A) and brain control 
session (B) Mean cross correlogram for the correct and error trial group shown in panel on right of each 
panel. The correlation is higher in correct trials indicative of increased synchronous neuronal modulations 
between monkeys. (C,D) k-NN prediction of trial outcome (reward or error) using the mean neuronal 

cross correlogram on a single trial (i.e. left panels in A-B) between the two monkeys either prior to target 
appearance (“Pre-target”) or after target appearance (“Post-target”). Chance level prediction (95% confidence 
interval) shown in yellow band. * denotes P < 0.01, unpaired t-test. Neuronal synchrony between monkeys 
before and after target onset was predictive of trial outcome. (E,F) Extra correlation analysis. Velocity profile 
in a trial for dyad M&O were cross-correlated. The average cross correlation (trial specific correlation, red 
trace) was estimated for all trials: hand control (E) and brain control (F) Extra correlation was the excess 
correlation (in the red trace) that cannot be accounted for by the distribution of across-trial correlation 
(enclosed by the grey dashed lines). The vertical line shows the time lag at peak correlation. (G-H) 
Partitioned control task: (G) Panels show the average trajectory to the 4 target locations, denoted by colour, 
from the first (left column) and the last (right column) session. Trajectories derived from the monkeys’ 
controlled axes predictions (controlled) are shown in the upper panels and non-controlled axes predictions 
(non-controlled) are shown in the lower panels. Non-controlled traces were initially more similar to the 
controlled traces (left panels) but over time became shrunk and convoluted (right panels). (H) The fraction 
of rewarded trials was computed based on trials when the avatar arm is moving according to the predictions 
of the controlled axis (blue) or the non-controlled axes (red). Mean fraction correct trials shown for each 

of the three weeks of experiments. Shown separately is fraction correct amongst all trials (upper panel) 

and only among trials where a reward was achieved (lower panel). The percentage of trials in which the 
complementary trace reached the target (red bars) decreased significantly over training. 
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effect of neural synchrony on the trial outcome was stronger in brain control mode (Fig. 3D). We suggest 
that the presence of neural correlation prior to target onset indicated that the monkeys attended to the 
avatar arm since the common visual input from the avatar arm was the only source of neural correlation 
during that period. This interpretation in terms of an increased attention agrees well with the better 
performance on the trials with high neural correlation: the more the monkeys attend to the avatar arm 
and the other screen events, the better they perform. 

Next, we examined whether the increase in concurrent neuronal activity between the two monkeys 
could emerge due to the fact that the two animals move simultaneously, without necessarily paying 
attention to each other, following visual stimulus onset. If this scenario were true, there would not be 
any trial specific neuronal correlation beyond the level that could be observed if the behavioural trials 
of one of the monkeys were randomly shuffled (provided that target locations still matched for two 
monkeys). To investigate this possibility, we used movement velocities as a proxy to test the correlation 
between monkeys. To determine the additional component to the coordination between the monkeys, 
over and above the coordination that comes about as a result of jointly performing reaching movements 
to a specific target location, we calculated a parameter called “extra-correlation”. Extra-correlation is the 
correlation between the two monkeys’ movement velocities in a trial that cannot be explained by the 
across-trial correlation. Across-trial correlation was computed by correlating velocity of movement of a 
monkey from one trial with that of the partner monkey on another randomly chosen trial in which the 
monkeys moved to the same target location. This was performed for several such combinations of trials 
(40-50 combinations) to obtain a distribution of the across-trial correlation between the velocities of the 
dyad. We observed that extra correlation in movement velocities, which was significant in 3/13 sessions 
during hand control (Fig. 3E), was however, significant during all brain control sessions for both dyad 
MO (Fig. 3F) and MC (Fig. S2). These results suggest that increases in concurrent activity is not only 
due to motor outputs triggered by the same visual stimulus, but also due to a significant trial specific 
coordination in movement velocities, especially during brain control. 


Experiment 2: Partitioned control. In the next experiment, a B2 (dyad M&C) performed a parti- 
tioned control task where each monkey performed a subtask of a 2D movement (Fig. 1C). One monkey 
controlled only the X position of the avatar arm (X-monkey) and the other monkey controlled only the Y 
position (Y-monkey). The targets appeared on the diagonals at 45°, 135°, 225°, or 315°, which meant that 
both the X-monkey and Y-monkey had to simultaneously and accurately enact movements along their 
individual axes to achieve a correct trial. Each session started with a period of passive observations of 
the avatar arm movements. Based on the data recorded during this period, one UKF was fit per monkey. 
During brain control, the UKF outputs (X for monkey M, and Y for monkey C) were combined. The 
unused outputs (Y for monkey M, and X for monkey C) were computed, but not shown to the monkeys 
as any kind of feedback. We compared the used and unused X and Y outputs in an offline analysis to 
estimate cortical adaptation to preferentially represent the coordinate being controlled through the BMI. 
During brain control mode, despite being given control of only one axis, a monkey could in princi- 
ple generate motor plans at the neural level that encoded movements in both the X and Y dimensions. 
Alternatively, each monkey could parse the 2D trajectory into a controlled dimension (which it had to 
attend to) and non-controlled dimension (which it could disregard). For example, if monkey M con- 
trolled movements along X-axis, we asked how this animal's cortical neurons changed their contribution 
to movements along Y-axis, which they did not control. That provided us with a way to quantify how 
much functional plasticity was occurring in the monkeys’ cortex as a result of partitioned Brainet oper- 
ation. To measure that, we first plotted the actual avatar movement traces generated by the B2 (Fig. 3G; 
upper panels) which corresponds to the axes the monkeys controlled (X for monkey M, and Y for mon- 
key C). We then compared this plot with the traces for the complementary pair of dimensions (Y for 
monkey M, and X for monkey C; Fig. 3G: lower panels). We found that the members of the B2 contrib- 
uted more along the dimension they controlled and less along the dimension they did not control. This 
result is evident from the comparison of the actual avatar movement traces generated by the B2 (Fig. 3G; 
upper panels) to the complementary traces (Fig. 3G; lower panels). These traces were similar in the early 
sessions (Fig. 3G; left panels) but diverged over time (Fig. 3G, right panels). By the 3"? week of training, 
average complementary traces were shrunk and convoluted compared to the average actual traces. As a 
result of this cortical adaptation, the complementary trajectories seldom reached the target. The percent- 
age of trials in which the complementary trace reached the target (red bars in Fig. 3H) decreased from 
26% to 12% over training (regression, r*= 0.42 upper panel of Fig. 3H, P < 0.05). Similar reduction was 
observed when only the rewarded trials were included (1? = 0.38, P < 0.05, lower panel Fig. 3H). 


Experiment 3: Brainet triad control of an avatar arm in 3-D space. In the third experiment, we 
built a 3D super-task out of individual 2D BMI tasks. Each of the three monkeys forming a B3 viewed the 
2D projection of a spherical target in 3D space, from an X-Y, Y-Z, or X-Z reference frame (Fig. 1D). The 
monkeys moved the avatar arms on their displays in their 2D space to the projected target location. As 
they reached the projected target, a 3D avatar arm that received inputs from all the three UKF decoders 
reached the true target location in 3D space (Movie S3). This design reduced task difficulty for any one 
individual monkey by breaking down the 3D task into simpler 2D subtasks. 
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Figure 4. Triad (B3) control of 3D movements. (A) Boxplots comparing target acquisition time (left 
panel), trial duration (centre panel), inter-reward interval (right panel). (B) Reduction in trial duration (left 
panel) and concurrent increase in the reward rate (right panel) with conjoint training across weeks. (C) 
Normalized contribution of each of the three monkeys across a representative subset of 30 trials. The relative 
contribution of each monkey varied from trial to trial. (D) Fraction of trials that were correctly performed 
by a dyad (black) as a triad (green), or incorrectly (purple) shifted across the 11 triad experiments. The 
fraction of total trials with a rewarded outcome in which all three monkeys contributed (green), or those in 
which two monkeys contributed (black) increased significantly within each week and across sessions whereas 
the fraction of erroneous/unattempted trials reduced significantly. (E,F) Decoded trajectories and neural 
data from the triad experiment. (E) Mean X,Y,Z traces produced by individual monkeys shown separately 
by colour among trials where all monkeys contributed (left column) or when only monkeys M and C 
contributed (right column). Mean X,Y,Z (the value used to move avatar) shown in black. Distance to target 
in each axis is 5 screen-cm. When one monkey (monkey K) opted out, the working dyad generated higher- 
amplitude trajectories (Right column, X axis and Z axis) as opposed to when all the members contributed 
(left column). (F) PETHs aligned on target onset for same trial subsets as in (E) Rows represent individual 
neurons and colour indicates normalized firing rate (z-score). Neurons from different monkeys marked 

by colour along right edge (same colours as in (E)). Increased effort by the working dyad also resulted in 
stronger cortical modulations between the members (right panel) as compared to when all the members 
contributed (left panel). 
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Each session started with a period of passive observation during which each monkey watched the 
computer-generated avatar arm perform the corresponding subtask. A UKF model was fit per monkey 
and its subtask, and the individual monkey outputs (X, Y for monkey M; Y, Z for monkey C; and X, Z for 
monkey K) were combined to produce 3D movement of the avatar arm. Put another way, during brain 
control, pairs of monkeys shared equal control over one of the axes: Monkeys M and K controlled the 
X-axis, monkeys M and C controlled the Y-axis, and monkeys C and K controlled the Z-axis. 

Clear behavioural improvements occurred during the brain control epochs over a span of three 
weeks of training. We observed a significant reduction in target acquisition time (p < 0.02; KS test), trial 
duration (p< 0.03; KS test) and inter-reward interval (p< 0.002; KS test) between early and late ses- 
sions (Fig. 4A). The mean trial duration significantly decreased (4.25 to 3.65s, 1-way ANOVA: p < 0.05, 
Fig. 4B panel on the left) over the span of three weeks and the mean reward rate increased from 6 to 
10 trials per minute (Fig. 4B, panel on the right). Across 11 sessions, the B3 significantly improved its 
performance from 20% correct trials in the first session to 78% correct trials in the last session (P < 0.01, 
1-way ANOVA with bootstrapping, green+black bars in Fig. 4D). 

Table 1 shows the overall mean target acquisition time, target hit rate and the movement precision 
required for the B3. Due to the addition of the third dimension, movement precision required of B3 was 
scaled by a factor of 5.5 as compared to B2. However, the target acquisition times increased only by a 
factor of 1.9 (Dyad MC vs. Triad MCK). Target hit rate remained nearly the same for both paradigms. 
The increase in target acquisition times indicates the cost of synchronizing an additional monkey into a 
B3. However, as a result of the design, the B3 performed reach movements in conditions that required 
greater movement precision, without compromising the hit rate. 

Even though the relative contribution of each monkey varied from trial to trial (Fig. 4C), the highest 
rate of success was attained when all three monkeys contributed (Fig. 4D). Furthermore, the percent 
of total trials with a rewarded outcome in which all three monkeys contributed to the final movement 
outcome (green bars in Fig. 3D) grew from 7% to 50%, the largest increase of any possible dyad or 
triad tested in this study. The increase was significant within each week of training (post-hoc Tukey test: 
P< 0.05) and across weeks (1-way ANOVA with bootstrapping: P < 0.001). 

The B3 design was resilient to any one individual monkey underperforming in a given trial because 
any monkey dyad could successfully move the avatar arm to the true target location in 3D space. We 
observed that the number of rewarded trials for the dyad M&C was lower as compared to the triad but 
improved with sessions as well (from 12% to 27%, P<0.01, 1-way ANOVA with bootstrapping). This 
improvement suggests that the B3 could benefit from the redundancy that was built into the design in the 
form of shared control. However, when one monkey opted out, the working dyad would need to produce 
higher-amplitude trajectories (Fig. 4E, right column) as opposed to when all the members contributed 
(Fig. 4E, left column). Increased effort by the working dyad also resulted in stronger cortical modulations 
between the members (Fig. 4F, right panel) as compared to when all the members contributed (Fig. 4F, 
left panel). 


Neuron-dropping curves: decoding improves with number of neurons. During B2 and B3 
operations, cortical ensembles in each monkey exhibited clear task-related activity during both passive 
observation (Fig. 5A,C) and brain control modes (Fig. 5B,D). Analysis of these neural signals confirmed 
that the accuracy of arm movement decoding would improve when VLSBA was recorded and combined 
from multiple brains (Fig. 5C). This finding extends our previous results, using neuron dropping curve 
(NDC) analyses, where we showed that decoding accuracy consistently improved when larger neuronal 
samples were recorded from a single brain*». Here, this analysis has been extended to visualize the effect 
of different relative quantities of two and three brains performing together as part of a Brainet. NDCs 
were constructed for the decoding of avatar arm position during passive observations and brain control 
mode for the B2 (Fig. 5E,F) and B3 (Fig. 5G-I). NDCs were plotted as families of curves, where each 
curve represented decoding accuracy for a neuronal sample composed of a variable-size ensemble from 
one monkey (Mk1) and a fraction of the full ensemble from the other monkey (Mk2). The NDCs indi- 
cated that decoding accuracy benefited from mixing the contributions from different brains as well as 
the overall neuronal mass. The best accuracy was typically achieved when all neurons from all monkeys 
were combined (Fig. 5E-I), with the only exception being the Z-axis prediction in B3 (Fig. 51). When 
only a small number of Mk1 cells (fewer than 10) were added to Mk2 ensembles during passive obser- 
vation, decoding accuracy stayed the same or slightly decreased (Fig. 5E-I). However, by scaling to 10? 
or 10° neurons, this trend shifted such that additional Mk1 neurons added to Mk2 ensembles yielded 
more accurate predictions. 

The NDC analysis indicated that Brainet decoding accuracy improved with an increase in the number 
of neurons. However, a larger total number of neurons was not the sole factor that contributed to the 
improvement, particularly, the improvement in performance over time. The total number of recorded 
neurons per monkey did not change considerably throughout the study (Monkey M: 410-501; Monkey 
C: 196-214; Monkey O: 156-229; Monkey K: 140-156). Yet, we observed a significant improvement in 
performance (Fig. 2A,B, Fig. 4A,B), as well as improvement in coordinated behaviour (Fig. 2I) and 
occurrence of concurrent neuronal activity (Fig. 2J). Coordinated behavior was observed even during 
hand control when the recorded neurons did not influence behavior. 
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Figure 5. Neuronal representations during B2 and B3 shared control experiments. Neuronal modulations 
from monkey M (orange) and C (green) during a 10 second window of passive observation. (A) and brain 
control (B) Centre and peripheral target onset times are denoted by grey vertical lines. Cortical ensembles in 
each monkey exhibited clear task-related activity during both passive observation and brain control modes. 
(C) Passively observed trajectory (black) compared with predicted trajectory using only Monkey M (green), 
only Monkey C (red), or both (blue) neuronal ensembles. Grey vertical lines from (A) again denote relevant 
task events. Accuracy of arm movement decoding improved when VLSBA was recorded and combined from 
multiple brains. (D) Decoded X and Y trajectories from monkey M (black, solid) and monkey C (red), as 
well as the average of the two (dotted) during 8 second window of brain control experiment. (E,F) Neuron 
dropping curves (NDC) showing effect of ensemble size of each of the two monkeys on prediction accuracy 
during B2 passive observation. The number of neurons used from Mk1 marked by x-axis. The percent of 
Mk2 population used for predictions denoted by colour (see Legend). Accuracy of predictions measured 

as correlation coefficient r. The decoding accuracy benefited from mixing the contributions from different 
brains as well as the overall neuronal mass. (G-I) Same NDC analysis as (E,F) except for prediction of X, Y, 
and Z position during B3 passive observation. Prediction of X (G), Y (H), and Z (I) shown separately. Again 
(as seen in E-F), the decoding accuracy benefited from mixing the contributions from different brains as 
well as the overall neuronal mass. 
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Additionally, improvements in Brainet performance were related to conjoint behavior of the monkeys 
(as in team work) rather than improvements in individual skills or individual decoding accuracy. All 
monkeys involved in these experiments had been previously overtrained on the single-BMI task, which 
makes individual improvements in BMI control an unlikely explanation for the overall improvement 
of the Brainet performance. Furthermore, we assessed the decoding accuracy in individual monkeys 
across sessions of manual control (Fig. 6A) and passive observations (Fig. 6B-D). The decoding accu- 
racy showed some fluctuations, which is typical as the motivation levels vary between days of the week. 
However, there was no systematic trend overall, which contrasts to a steady improvement in Brainet 
performance over time. These observations suggest that coordinated behavior rather than individual 
improvements the major factor, which led to better performance. 


Discussion 

We have successfully developed a shared monkey BMI that utilized chronic, simultaneous VLSBA 
recordings from pairs or triplets of primate brains to enact conjoint 2D and 3D movements of a monkey 
avatar arm. This shared BMI, or Brainet, has scaled the conventional definition of BMI from a technol- 
ogy that derives neuronal signals from a single brain to one that can functionally link multiple brains to 
coordinate movements in space and time in an unsupervised way. 

Previously, joint motor behaviour was studied extensively in human psychophysics studies '*!?*!. Two 
individuals who are performing movements can become entrained when they mutually affect each other’s 
behaviour”, even if they are not explicitly instructed to do so”. In our study, we have shown evidence for 
both behavioural and neurophysiological entrainment in nonhuman primates that are interacting, indi- 
rectly, via a common avatar limb in a virtual environment. The avatar arm provided the same individual 
action opportunity for all participants involved - also known as simultaneous affordance”!*. Optimal 
performance has been proposed to occur in conjoint action when participants can monitor others’ move- 
ments” which was facilitated in our experiment through common visual feedback. 

The various Brainet configurations tested in this study revealed different types of behavioural and 
neurophysiological adaptations. Individual monkeys in a B2 could optimize the components of the arm 
movement they controlled directly over the non-controlled components. At the same time, the dyad 
adapted spatial components (Fig. 3A,B) while maintaining temporal behaviour/brain coordination. These 
results suggest that neurophysiological adaptations may be context-dependent. 

Shared BMI control introduced redundancy and computational complexity into the Brainet’s oper- 
ation by allowing the final motor goal to be achieved despite occasional suboptimal behaviour by an 
individual monkey. Furthermore, as observed in some recent EEG studies***4, the NDCs have demon- 
strated that merging neuronal populations from two/three brains are optimal from the perspective of 
decoding accuracy. 

Another important aspect of our Brainet design is the ability to compensate for performance fluc- 
tuations in individual members. For example, in our B3 experiments, we often noticed that monkey C 
compensated for monkey K’s lack of contribution to movements along the y-axis (Fig. 5D). These obser- 
vations highlight the potential of a Brainet in assisting individual members to overcome their limitations 
with the help of the partners. 

In conclusion, this study has successfully established a general paradigm for establishing a closed 
loop shared BMI, formed by two or more brains, through visual feedback. Shared BMI allowed multiple 
monkey brains to adapt in an unsupervised manner. Based on these evidences, we propose that primate 
brains can be integrated into a self-adapting computation architecture capable of achieving a common 
behavioural goal. 


Methods 

Subjects and implants. All studies were conducted with approved protocols from Duke University 
Institutional Animal Care and Use Committee and were in accordance with NIH Guidelines for the 
Use of Laboratory Animals. Four adult rhesus macaque monkeys (Macaca mulatta) participated in this 
study. Monkey M and K were chronically implanted with four and six 96 channel arrays, respectively, 
in bilateral arm and leg areas of M1 and S1 cortex. Monkey C was implanted with eight 96 channel 
multielectrode arrays in bilateral M1, S1, dorsal premotor (PMd), supplementary motor area (SMA), 
and posterior parietal cortex (PPC)**. In Monkeys M, C, and K, the multielectrode array consisted of a 
4x 8 (M) or 4x 10 (C,K) grid of cannulae containing microwire bundles which enable different depths 
of cortical tissue to be sampled. Microelectrode array design and surgical procedure has been previously 
described 35364, Monkey O was implanted with four 448-channel multielectrode arrays, one each in left 
and right hemisphere M1 and S1°*°. In each of these arrays, shafts that contained the microwires were 
arranged in a 10 x 10 layout. Each shaft consisted of 3-4 microwires with 500j1m tip spacing or 6 -7 
microwires with 1000,1m tip spacing. For experiments in this study, the recordings in were drawn from 
M1 (all monkeys) and S1 (only monkey M) cortical areas. 


Experimental Setup. Prior to experiments of this study, each monkey was trained to perform 
centre-out reaching movements using their left arm to move a spring-loaded joystick. For hand con- 
trol versions of shared control, both monkeys moved the joystick. For all brain control experiments, 
the upper limbs of each participating monkey were gently restrained using an established technique*. 
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Figure 6. Decoding accuracy in individual monkeys. A UKF decoder trained post hoc predicted 
monkey’s own movements in every session. (A-C) Upper panels show the performance of the decoder 

for the x-component of the trajectories and lower panels for y-component of the trajectories. Each data 
point represents the average performance for the week. Correlation coefficient r was utilized to measure 
the accuracy of predictions. The performance of the decoder was compared across weeks to monitor the 
changes in the contribution of the neuronal population during the course of training in the shared control 
task (A,B): manual control in (A) and passive observation epoch preceding brain control in (B) and the 
partitioned control task (C). (D) shows the decoder performance for the x-, y- and z- components of the 
trajectories (top, middle and bottom panels, respectively) in the triad task. Overall, the performance of the 
decoder remained stable and no consistent trends were observed. 
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Approximately 50cm in front of each monkey, at eye level, a computer monitor (26cm x 30cm.) dis- 
played the virtual avatar limb*”® and target objects from a first-person perspective. The virtual environ- 
ment was created using Motionbuilder software (Autodesk, Inc.). In all configurations of the task, each 
participating monkey was placed into a different experiment room with no visual or auditory contact 
with the other monkeys. 

During experiments, neural activity was recorded from each monkey using commercially available 
Plexon systems (Plexon, Dallas, TX). Spike timestamp information from each of the four (for dyad 
experiments) or five (for triad experiments) recording systems was sent over the local network to a 
master computer for neural decoding steps. The master computer contained the BMI software suite, 
which controlled task flow, decoder input/output, and reward delivery. Once kinematic predictions were 
generated by the BMI, the avatar movement commands were sent over UDP packets to Motionbuilder in 
order to update the position of the avatar limb being displayed on the screens. In all dyad experiments, 
the two monkeys were provided identical visual feedback. In the triad experiment, the visual feedback 
was different for each monkey. 


Dyad Task Paradigms. Shared control task. A circular target (5cm diameter) appeared in the centre 
of the screen. The monkeys must hold the avatar arm within the centre target for a random hold time 
uniformly drawn from the interval 500-1200 ms. Next, the central target disappeared and was replaced 
by a circular disc (5cm. in diameter) in one of 4 peripheral locations (left, right, above, or below the 
centre-point on the screen). The centre of the target disc was 9cm from the centre. When the monkeys 
placed the arm within the target such that the cursor (0.5cm. diameter) at the base of the middle finger 
(Fig. 1C) was completely within the target, for a hold interval (500 ms), the trial was considered correct 
and a small juice reward was dispensed. Note that the cursor is shown for illustration purposes; it was not 
visible to the monkey. Trials where the target was not reached within 10 seconds (hand control) or 15 sec- 
onds (brain control) were considered error trials, resulting in no juice and a 1 second timeout period. 

The position of the avatar arm was exactly the average of the position derived from each of the two 
monkeys. In hand control, this was the mean (X,Y) joystick position. For brain control experiments, 
the neural decoder was fit using 5-6minutes of data collected during passive observation. The neural 
decoding algorithm used throughout this study was the unscented Kalman filter (UKF)*”. During brain 
control, the avatar hand (X,Y) position was decoded from each monkey’s brain individually, then aver- 
aged. Next, the averaged position was used to update the avatar hand position, thus providing the visual 
feedback to both monkeys. 


Passive observation. During this task mode, each of the two monkeys in the dyad passively observed 
identical, pre-programmed reach trajectories on the screen**. The hand moved along the ideal trajectories 
between centre and peripheral target starting ~500 ms after target onset. The observed hand accelerated 
and decelerated in a realistic manner when starting and stopping a reach. Juice rewards were dispensed 
when the hand reached and held inside the displayed target. This mode was used for 6-7 minutes at the 
beginning of all brain control experiments. 


Partitioned control task. All sessions began with 5-6 minutes of passive observation trials. In this task, 
all targets appeared at the diagonal locations: 45°, 135°, 225°, and 315° in Cartesian space. Both the mon- 
keys viewed the hand move along the ideal trajectories between centre and peripheral target. Centre and 
peripheral target sizes, the distance of the target from the centre were the same as in the previous task. 
Following passive observation, one UKF model was fit for each monkey. The UKF models for monkeys 
M and C were used to decode only X or only Y, respectively. Next, the task was switched to brain control. 
Monkey M was given full control of the X position of the avatar hand, and monkey C was given full 
control of Y position. Together the dyad had to reach toward the target to complete the trial successfully. 
The pair had to work together to move the hand to the centre, hold briefly, and then move out to the 
peripheral target, then hold. The hold time and reward contingencies were retained from the previous 
task. A total of 8 such experiments were completed over a span of three weeks. 


Triad Task Paradigm. Monkeys M, C, and K formed a Brainet triad to generate 3D reaching move- 
ments of the virtual avatar arm. Once again, centre-out movements were performed. Here the peripheral 
target was located at one of the 8 equidistant corners of a 3D cube. The projected target was 5cm. in 
diameter and its centre was located 7cm from the centre of the screen. Each monkey observed only a 
2D projection of this movement on their display screen. Monkeys M, C and K were shown the target 
from an X-Y, Y-Z, or X-Z reference frame respectively. The monkey viewed the avatar arm always from 
a first-person perspective. 

Sessions began with 6-7 minutes of passive observation trials viewed from each monkey’s designated 
reference frame. Three UKF decoders, one for each monkey, were then initialized and fit on the passive 
observation data. Each UKF was trained to decode the two dimensions that could be viewed on that 
monkey’s screen (e.g. X and Y for monkey M). Thus, two UKF models represented X, two represented 
Y, and two represented Z. Pairs of monkeys with a common dimension were given 50% control of that 
axis. More specifically, Monkeys M and K each had 50% control of the X axis, Monkeys M and C each 
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had 50% control over the Y axis, and Monkeys C and K each had 50% control of the Z axis. The decoded 
and averaged X, Y, and Z positions derived from the Brainet triad were then used as a control signal for 
resultant 3D movements. 2D movement projections were shown as visual feedback to each monkey. For 
analysis purposes, we considered a monkey to have contributed in a trial if the movements generated by 
that monkey covered at least 25% of the total distance. 


Analyses. All the analyses were performed in Matlab (Mathworks Inc.) using built-in functions, 
open-source code, or custom-designed analysis tools developed within the Duke University Center for 
Neuroengineering. 


Analysis of neuronal activity. Neurons that were not active for greater than 50% of the session duration 
were considered quiet neurons and were removed. For the rest, recorded action potential events were 
counted in bins of 50 ms width and aligned on target onset to obtain PETHs. PETHs were typically calcu- 
lated for each trial and then were averaged across trials. The average modulation profile for each neuron 
was normalized by subtracting the mean bin count and dividing by the standard deviation of the cell’s 
bin count; both values were calculated for raw spike trains prior to any PETH calculations. With this 
normalization, PETHs express the event-related modulations as a fraction of the overall modulations, or 
statistically, the z-score. 


Estimating reaction time lag. The joystick positions (in hand control) or the UKF outputs (in brain con- 
trol) of the two monkeys were cross-correlated after subtracting the data mean and normalizing to a -1 to 
1 scale. The position of the peak of the cross-correlation relative to the ordinate provided an estimate of 
the time lag between the dyad members. Lag was calculated for every rewarded trial and then averaged 
across all trials to get an estimate of the average lag between the dyad members in a session. 


Estimating neural activity lag. Temporal relationships between the neural activity of monkey dyads was 
determined using a cross-correlation with the mean subtracted and normalized to set the auto-covariance 
terms equal to 1. This allowed the cross-correlations between different trial conditions to be comparable. 
To compute the between-monkey lag, the single trial PETHs (spike counts within 50ms bins during a 
fixed —0.5 to 1s window relative to target appearance) were collected for each neuron on each trial. All 
pair wise combinations between Monkey 1 cells and Monkey 2 cells were cross-correlated. This yielded 
a mean cross-correlogram for each trial, as shown in the left panels in Fig. 2G,H. The time of the peak 
of the cross-correlogram on each trial was identified and defined to be the single-trial lag between the 
two monkeys. Mean lag for a session was computed by averaging across trials (Fig. 2F). 


Trial duration. ‘Trial durations were calculated from the target onset time to when the reward was 
delivered. To avoid potential confounds due to satiation effects, mean/ average trial duration (Fig. 2C,D, 
Fig. 4B) was estimated during the initial 15 minutes of every session when the motivation was highest. 
However, trial durations shown in Fig. 2A,B and Fig. 4A include all trials in the session. 


k-NN prediction of trial outcome. To predict trial outcome based on level of between-monkey corre- 
lation, single trial PETHs were computed for each neuron on each trial. The “pre-target” window was 
defined as from 700 ms before target appearance until target appearance. The “post-target” window was 
defined as from 100 to 800ms after target appearance. The activity profile of each neuron during the 
specified epoch was normalized and then cross-correlated with the activity profile from each neuron 
from the other monkey. The cross correlogram for each pairwise neuronal combination was then aver- 
aged to produce a single between-monkey cross correlogram for a given trial. The set of single-trial cross 
correlograms were then used to predict the outcome of their corresponding trials or that of a randomly 
chosen trial (control). 80% of trials were designated to be training data and 20% of trials were test data. 
A k-nearest neighbour classifying algorithm (k-NN with k= 5) was fit and then tested by predicting trial 
outcome on the designated test trials. Performance was quantified in terms of fraction correct prediction. 
Training/test data was redrawn 10 times. We reported the distribution of the fraction correct values after 
redrawing the training/test data (Fig. 21,J). 


Neuron dropping analysis. Neural activity was recorded and binned into 100ms bins during passive 
observation experiments. An unscented Kalman filter (UKF) with three 100 ms past taps of neural activ- 
ity and two 100 ms future taps of neural activity was used to decode the X and Y position of the observed 
avatar hand on the screen. UKF prediction of X and Y were computed for one of the monkeys for a wide 
range of neurons, ranging from 1 to the full ensemble, in increments of 3 (n< 20), 7 (at 20<n< 100), or 
15 (n> 100). The predicted X and Y traces were compared with the actual X and Y traces, the correlation 
coefficient r was computed for each, and then averaged. In B2 configurations, a second monkey was then 
added to the analysis. X and Y UKF predictions using 25%, 50%, 75% and 100% of the 2"? monkey’s 
neurons were then computed. The predicted X and Y were averaged with the X and Y from Monkey 
1’s prediction, as was done during on-line experiments. The accuracy of these averaged predictions was 
quantified using correlation coefficient as well. 
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In B3 configurations, the predictions were generated in a similar way as for B2. The two coordinates 
controlled by each monkey (X-Y, Y-Z, or X-Z) were likewise decoded from the ensembles of varying 
sizes using a UKF decoder. For X, Y, and Z, predictions were averaged across the two monkeys who 
shared each axis. For example, the X predictions from Monkey M and K were computed and averaged 
to produce one r value for each of the different quantities of Monkey M and K neurons. Fixed percent 
quantities of the second monkey were added to the first monkey’s ensemble to show the effect of adding 
neurons from different subjects in a Brainet. 


Normalized contribution. In a trial, the fraction of the total distance moved by each monkey in the B3 
was computed along each axis. A monkey’s normalized contribution (Fig. 4C) was calculated by sum- 
ming its contributions (in the two directions) and normalizing it by dividing by 3. 

Negative values (capped at -0.2) indicate movements generated in the direction away from the target. 


ANOVA on bootstrapped samples. All the trials from a session were resampled several times with 
replacement and the fraction of error trials, rewarded trials contributed by all the members of the triad, 
or by a dyad was computed for each sample. A one-way ANOVA was performed on these samples across 
sessions. F value of the original sample was compared with those of the bootstrapped samples. 
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Recently, we proposed that Brainets, i.e. networks formed by multiple animal brains, cooperating 

: and exchanging information in real time through direct brain-to-brain interfaces, could provide 

: the core of a new type of computing device: an organic computer. Here, we describe the first 

: experimental demonstration of such a Brainet, built by interconnecting four adult rat brains. Brainets 
worked by concurrently recording the extracellular electrical activity generated by populations 
of cortical neurons distributed across multiple rats chronically implanted with multi-electrode 

: arrays. Cortical neuronal activity was recorded and analyzed in real time, and then delivered to 

: the somatosensory cortices of other animals that participated in the Brainet using intracortical 
microstimulation (ICMS). Using this approach, different Brainet architectures solved a number 
of useful computational problems, such as discrete classification, image processing, storage and 

: retrieval of tactile information, and even weather forecasting. Brainets consistently performed at 

: the same or higher levels than single rats in these tasks. Based on these findings, we propose that 

: Brainets could be used to investigate animal social behaviors as well as a test bed for exploring the 
properties and potential applications of organic computers. 


After introducing the concept of brain-to-brain interfaces (BtBIs)', our laboratory demonstrated exper- 
imentally that BtBIs could be utilized to directly transfer tactile or visuomotor information between 
pairs of rat brains in real time’. Since our original report, other studies have highlighted several proper- 
: ties of BtBIs'?, such as transmission of hippocampus representations between rodents’, transmission of 
: visual information between a human and a rodent?, and transmission of motor information between two 
: humans®’, Our lab has also shown that Brainets could allow monkey pairs or triads to perform coop- 
erative motor tasks mentally by inducing, accurate synchronization of neural ensemble activity across 
individual brains’. 

In addition to the concept of BtBIs, we have also suggested that networks of multiple interconnected 
animal brains, which we dubbed Brainet', could provide the core for a new type of computing device: an 
organic computer. Here, we tested the hypothesis that such a Brainet could potentially exceed the perfor- 

: mance of individual brains, due to a distributed and parallel computing architecture!*. This hypothesis 
: was tested by constructing a Brainet formed by four interconnected rat brains and then investigating 
: how it could solve fundamental computational problems (Fig. 1A-C). In our Brainet, all four rats were 
chronically implanted with multielectrode arrays, placed bilaterally in the primary somatosensory cortex 
(S1). These implants were used to both record neural ensemble electrical activity and transmit virtual 
tactile information via intracortical electrical microstimulation (ICMS). Once animals recovered from 
the implantation surgery, the resulting 4-rat Brainets (Fig. 1) were tested in a variety of ways. Our central 
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Figure 1. Experimental apparatus scheme for a Brainet computing device. A) A Brainet of four 
interconnected brains is shown. The arrows represent the flow of information through the Brainet. Inputs 
were delivered as simultaneous ICMS patterns to the $1 cortex of each rat. Neural activity was then recorded 
and analyzed in real time. Rats were required to synchronize their neural activity with the remaining of 

the Brainet to receive water B) Inputs to the Brainet were delivered as ICMS patterns to the left $1, while 
outputs were calculated using the neural responses recorded from the right $1. C) Brainet architectures were 
set to mimic hidden layers of an artificial neural network. D) Examples of perievent histograms of neurons 
after the delivery of ICMS. 


goal was to investigate how well different Brainet architectures could be employed by the four rats to 
collaborate in order to solve a particular computational task. Different Brainet designs were implemented 
to address three fundamental computational problems: discrete classification, sequential and parallel 
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Figure 2. The Brainet can synchronize neural activity. A) The different colors indicate the different 
manipulations used to study synchronization across the network. During the pre-session, rats were tested 

for periods of spurious neural synchronization. No ICMS or rewards were delivered here. During sessions, 
rats were tested for increased neural synchronization due to detection of the ICMS stimulus (red period). 
Successful synchronization was rewarded with water. During the post session, rats were tested for periods 

of neural synchronization due to the effects of reward (e.g. continuous whisking/licking). Successful 
synchronization was rewarded with water, but no ICMS stimulus was delivered. B) Example of neuronal 
activity across the Brainet. After the ICMS there was a general tendency for neural activity to increase. Periods 
of maximum firing rate are represented in red. C) The performance of the Brainet during sessions was above 
the pre-sessions and post-sessions. Also, delivery of ICMS alone or during anesthetized states also resulted in 
poor performances. ** and *** indicate P< 0.01 and P< 0.0001 respectively. D) Overall changes in R values 
in early and late sessions show that improvements in performances were accompanied by specific changes in 
the periods of synchronized activity. E) Example of a synchronization trial. The lower panels show, in red, 
the neural activity of each rat and, in blue, the average of neural activity for the remaining of the Brainet. 
The upper panels depict the R value for the correlation coefficient between each rat and the remaining of the 
Brainet. There was an overall tendency for the Brainet to correlate in the beginning of the test period. 


computations, and memory storage/retrieval'. As predicted, we observed that Brainets consistently out- 
performed individual rats in each of these tasks. 


Results 

All experiments with 4-rat Brainets were pooled from a sample of 16 animals that received cortical 
implants from which we could simultaneously record the extracellular activity from 15-66 S1 neurons 
per Brainet (total of 2,738 neurons recorded across 71 sessions). 


Brainet for neural synchronization. Rats were water deprived and trained on a task that required 
them to synchronize their neural activity after an ICMS stimulus. A total of six rats were used in 12 ses- 
sions to run this first experiment. As depicted in Fig. 1A-C, the processing chain in these experiments 
started with the simultaneous delivery of an ICMS pattern to one of the S1 cortices of all subjects, then 
processing of tactile information with a single-layer Brainet, followed by generation of the system output 
by the contralateral S1 cortex of each animal. Each trial was comprised of four epochs: waiting (base- 
line), ICMS delivery, test, and reward. ICMS patterns (20 pulses at 22-26 Hz) were unilaterally delivered 
to the S1 of each rat. Neuronal responses to the ICMS were evaluated during the test period when S1 
neuronal ensemble activity was sampled from the hemisphere contralateral to the stimulation site (Figs. 
1D and 2A-E) (Fig. 2A-E). Rats were rewarded if their cortical activity became synchronized during 
the test period. The correlation coefficient R was used as the measure of global Brainet synchrony. Thus, 
R measured the linear correlation between the normalized firing rate of all neurons in a given rat and 
the average normalized firing rate for all neurons recorded in the remaining three rats (see Methods for 
details). If at least three rats presented R values greater or equal to 0.2, a trial was considered successful, 
and all four rats were rewarded. Otherwise no reward was given to any rat. Two conditions served as 
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controls: the pre-session, where no ICMS or water reward were delivered, and the post-session, where 
no ICMS was delivered but rats were still rewarded if they satisfied the correlation criterion (Fig. 2A). 

Behaviorally, rats remained mostly calm or immobile during the baseline period. After the ICMS 
pattern was delivered simultaneously to all animals, rats typically displayed periods of whisking and 
licking movements. A sample of $1 neuronal population activity during this period is shown in Fig. 2B 
(also see Fig. 1D for examples of individual neurons perievent histograms). Typically, after the delivery 
of ICMS, there was a sharp decrease in the neuronal firing rate of the neurons (~20 ms), followed by a 
sudden firing rate increase (~100ms). While the main measure of accuracy for this task was the degree 
in which cortical neuronal populations fired synchronously, it is important to emphasize that the build 
up of these ensemble firing patterns depended highly on how single $1 neurons modulated their firing 
rate as a result of electrical microstimulation. Thus, ICMS served as a reset signal that allowed rats to 
synchronize their neural activity to the remaining network (Fig. 2D,E). Note that, in this task, rats were 
not exchanging neural information through the BtBI. Instead the timing of the ICMS stimulus, the par- 
tial contact allowed through the Plexiglas panels, and the reward were the only sources of information 
available for rats to succeed in the task. 

As the Brainet consistently exhibited the best performance during the first trials, we focused our 
subsequent analysis on the first 30-trial block of each session. Overall, this 4-rat Brainet was able to syn- 
chronize the neural activity of the constituent rats significantly above Pre-Session (Brainet: 57.95 + 2%; 
Pre-Sessions: 45.95 +2%; F,,= 10.99; P=0.0004; Dunnett’s test: P< 0.001) and Post-Session levels 
(46.41 + 2%; Dunnett’s test: P< 0.01; Fig. 2C). 

Over approximately 1.5 weeks (total of 12 sessions), this Brainet gradually improved its performance, 
from 54.76 + 3.16% (mean + standard error; the first 6 days) to 61.67 + 3.01% correct trials (the last 6 
days; F,,=5.770, P=0.0175 for interaction; Bonferroni post hoc comparisons: pre vs session initial start 
P > 0.05; pre vs session end P < 0.01; session vs post start P > 0.05; session vs post end P< 0.001). The 
high fidelity of information transfer in this Brainet configuration was further confirmed by the observa- 
tion that the performance of individual rats reached 65.28 + 1.70%. In other words, a 4-rat Brainet was 
capable of maintaining a level of global neuronal synchrony across multiple brains that was virtually 
identical to that observed in the cortex of a single rat (Brainet level = 61.67 + 3.07%; Man-Whitney 
U=58.0; P= 0.4818, n.s.). 

A comparison of correlation values between sessions from the first (n = 6) and the last days (n= 6) fur- 
ther demonstrated that daily training on this first task resulted in a statistically significant increase in cor- 
related cortical activity across rats, centered between 700 ms and 1000 ms of the testing period (F = 1.622; 
df= 1.49; P=0.0043, Fig. 2D). The lower panel of Fig. 2E shows the normalized firing rate for each rat (in 
red) and for the remaining Brainet (in blue) in one trial. The upper panels show R value changes for the 
correlation between neuronal activity in each rat and the remaining Brainet. Notice the overall tendency 
for most rats to increase the R values soon after the delivery of the ICMS pattern (T = 0 seconds). 

To determine if reward was mandatory for the correlation to emerge in the Brainet, we performed 
three control sessions with awake animals receiving ICMS (but no reward). The performances dropped 
to levels below chance (performance: 30.67 + 3.0%; see Fig. 2C). Further, in another three sessions where 
ICMS was applied to anesthetized animals, the Brainet performed close to chance levels again (perfor- 
mance: 38.89 + 4.8%; see Fig. 2C). These results demonstrated that the Brainet could only operate above 
chance in awake behaving rats in which there was an expectation for reward. 

After determining that the Brainet could learn to respond to an ICMS input by synchronizing its 
output across multiple brains, we tested whether such a collective neuronal response could be utilized 
for multiple computational purposes. These included discrete stimulus classification, storage of a tactile 
memory, and, by combining the two former tasks, processing of multiple tactile stimuli. 


Brainet for stimulus classification. Initially, we trained our 4-rat Brainet to discriminate between 
two ICMS patterns (Fig. 3A,B, 8 sessions in 4 rats). The first pattern (Stimulus 1) was the same as in the 
previous experiment (20 pulses at 22-26 Hz), while the second (Stimulus 2) consisted of two separate 
bursts of four pulses (22-26 Hz). The Brainet was required to report either the presence of Stimulus 1 
with an increase in neuronal synchrony across the four rat brains (ie. R> 0.2 in at least three rats), or 
Stimulus 2 by a decrease in synchrony (i.e., R< 0.2 in at least three rats). By requiring that the delivery 
of Stimulus 2 be indicated through a reduction in neuronal synchronization, we further ensured that the 
Brainet performance was not based on a simple neural response to the ICMS pattern. As in the previous 
experiment, Stimulus 1 served as a reset signal that allowed rats to synchronize their neural activity to the 
remaining network. Meanwhile, because Stimulus 2 was much shorter than Stimulus 1, it still induced 
neural responses in several S1 neurons (Fig. 3B), but its effects were less pronounced and not as likely to 
induce an overall neural synchronization across the Brainet (see Supplementary Figure 1). 

Following training, the Brainet reached an average performance of 61.24+0.5% correct discrimi- 
nation between Stimuli 1 and 2, which was significantly above No-ICMS sessions (52.97 + 1.1%, n=8 
sessions; Brainet vs No-ICMS: Dunn’s test: P < 0.01). Moreover, using this more complex task design, the 
Brainet outperformed individual rats (55.86 + 1.2%) (Kruskal-Wallis statistic = 10.87, P = 0.0044; Brainet 
vs Individual Rats; Dunn’s test: P < 0.05; also see Fig. 3C). 

To improve the overall performance of this 4-rat Brainet, we implemented an adaptive decoding 
algorithm that analyzed the activity of each neuron in each specific bin separately, and then readjusted 
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Figure 3. The Brainet can both synchronize and desynchronize neural activity. A) Architecture of 

a Brainet that can synchronize and desynchronize its neural activity to perform virtual tactile stimuli 
classification. Different patterns of ICMS were simultaneously delivered to each rat in the Brainet. Neural 
signals from all neurons from each brain were analyzed and compared to the remaining rats in the Brainet. 
The Brainet was required to synchronize its neural activity to indicate the delivery of a Stimulus 1 and to 
desynchronize its neural activity to indicate the delivery of a Stimulus 2. B) Example of perievent histograms 
of neurons for ICMS Stimulus 1 and 2. C) The Brainet performance was above No-ICMS sessions, and above 
individual rats’ performances. * indicates P < 0.05; ** indicates P< 0.01; n.s. indicates non significant. 


the neuronal weights following each trial (see Methods for details). Figure 4A depicts this Brainet archi- 
tecture. Notice the different weights for each of the individual neurons (represented by different shades 
of grey), reflecting the individual accuracy in decoding the ICMS pattern. Figure 4B illustrates a session 
in which all four rats contributed to the overall decoding of the ICMS stimuli (the red color indicates 
periods of maximum decoding). Using this approach, we increased both the overall Brainet performance 
(74.18 + 2.2% correct trials; n=7 rats in 12 sessions) and the number of trials performed (64.17 + 6.2 
trials) in each session. The neuronal ensembles of this Brainet included an average of 50+ 43 neurons 
(mean + standard error). Figure 4C depicts the improved performance of the Brainet compared to that 
of the No-ICMS sessions (54.34 + 2.2% correct trials, n= 11 sessions) and the performance of individ- 
ual rats (61.28 + 1.1% correct trials, F= 26.34; df= 2, 56; P< 0.0001; Bonferroni post hoc comparisons; 
Brainet vs No-ICMS: P < 0.0001; Brainet vs Individual rats P < 0.0001). 

When rats were anesthetized (2 sessions in five rats) or trial duration was reduced to 10s (i.e. almost 
only comprising the ICMS and the test period - 2 sessions in four rats), the Brainet’s performance 
dropped sharply (anesthetized: 60.61 + 2.8% correct; short time trials: 62.57 + 3.14%). Once again, this 
control experiment indicated that the Brainet operation was not solely dependent on an automatic 
response to the delivery of an ICMS. 

Next, we investigated the dependence of the Brainet’s performance on the number of S1 neurons 
recorded simultaneously. Figure 4D depicts a neuron dropping curve illustrating this effect. According 
to this analysis, Brainets formed by larger cortical neuronal ensembles performed better than those 
containing just a few neurons’. 

The difference between the Brainet classification of the two stimuli during regular sessions and during 
those in which no-ICMS was delivered is shown in Fig. 4E. During the regular sessions stimulus classi- 
fication remained mostly in the quadrants corresponding to the stimuli delivered (lower left and upper 
right quadrants), while during the No-ICMS sessions the 4-rat Brainet trial classification was evenly 
distributed across all quadrants. 

As different rats were introduced to the Brainet, we also compared how neuronal ensemble encod- 
ing in each animal changed during initial and late sessions (the first three versus the remaining days). 


SCIENTIFIC REPORTS | 5:11869 | DOI: 10.1038/srep11869 5 


www.nature.com/scientificreports/ 


im\ Rat3 Rat4 n=12 sessions n=2 sessions 
n=13 n=27 n=8 eiaiae 
RK 


cece e066 Khe eoee 


Neurons “s\ 


G ce © @_: 
E 
Hidden 
Rat1 Rat3 Rat4 layer 1 
G G Input 40 20 30 40 50 60 
Neuron number - Brainet NoICMS Individual Anesthetized 
Stim1 Stim2 0 % Decoding 85 Rats 
: Original Brainet 
D : E Brainet No ICMS F 9 


19.78% 48.47% 24.86% 27.3% 
— ———— 


% Correct 


Brainet Class 
Brainet Class 


mean=50.4+3 neurons 


Me 2 Githannourene: Stim Stim2 Stim Stim2 
Figure 4. Brainet for discrete classification. A) Architecture of a Brainet for stimulus classification. Two 
different patterns of ICMS were simultaneously delivered to each rat in the Brainet. Neural signals from 
each individual neuron were analyzed separately and used to determine an overall classification vote for the 
Brainet. B) Example of a session where a total of 62 neurons were recorded from four different animals. 
Deep blue indicates poor encoding, while dark red indicates good encoding. Although Rat 3 presented the 
best encoding neurons, all rats contributed to the network’s final classification. C) Performance of Brainet 
during sessions was significantly higher when compared to the No-ICMS sessions. Additionally, because 
the neural activity is redundant across multiple brains, the overall performance of the Brainet was also 
higher than in individual brains. *** indicates P < 0.0001. D) Neuron dropping curve of Brainet for discrete 
classification. The effect of redundancy in encoding can be observed in the Brainet as the best encoding 
cells from each session are removed. E) The panels depict the dynamics of the stimulus presented (X axis: 1 
or 2) and the Brainet classifications (Y axis: 1 to 2) during sessions and No-ICMS sessions. During regular 
sessions, the Brainet classifications mostly matched the stimulus presented (lower left and upper right 
quadrants). Meanwhile, during No ICMS sessions the Brainet classifications were evenly distributed across 
all four quadrants. The percentages indicate the fraction of trials in each quadrant (Stimulus 1, vote 1 not 
shown). F) Example of an image processed by the Brainet for discrete classification. An original image was 
pixilated and each blue or white pixel was delivered as a different ICMS pattern to the Brainet during a 
series of trials (Stimulus 1 - white; Stimulus 2 - blue). The left panel shows the original input image and the 
right panel shows the output of the Brainet. 


Overall, there was a significant increase in ICMS encoding (initial: 59.67 + 1.4%, late: 65.08 + 1.2%, 
Mann-Whitney U= 281.0, P= 0.0344) and, to a smaller extent, in the correlation coefficients between 
neural activity of the different animals (initial: 0.1831 + 0.007, late: 0.2028 + 0.005, Mann-Whitney 
U=275.0, P=0.0153) suggesting that improvements in Brainet performances were accompanied by 
cortical plasticity in the S1 of each animal. 

To demonstrate a potential application for this stimulus discrimination task, we tested whether our 
Brainet could read out a pixilated image (N= 4 rats in n= 4 sessions) using the same principles demon- 
strated in the previous two experiments. Blue and white pixels were converted into binary codes (white 
- Stimulus 1 or blue - Stimulus 2) and then delivered to the Brainet over a series of trials. The right panel 
of Fig. 4F shows that a 4-rat Brainet was able to capture the original image with good accuracy (overall 
87% correct trials) across a period of four sessions. 


Brainet for storage and retrieval of tactile memories. To test whether a 3-rat Brainet could store 
and retrieve a tactile memory, we sent an ICMS stimulus to the S1 of one rat and then successively trans- 
ferred the information decoded from that rat’s brain to other animals, via a BtBI, over a block of four 
trials. To retrieve the tactile memory, the information traveling across different rat brains was delivered, 
at the end of the chain, back to the S1 cortex of the first rat for decoding (Fig. 5A). Opaque panels were 
placed between the animals, and cortical neural activity was analyzed for each rat separately. The archi- 
tecture of inputs and outputs of the 3-rat Brainet’s is shown in Fig. 5A, starting from the bottom shelf and 
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Figure 5. A Brainet for storage and retrieval of tactile memories. A) Tactile memories encoded as 

two different ICMS stimuli were stored in the Brainet by keeping information flowing between different 
nodes (i.e. rats). Tactile information sent to the first rat in Trial 1 (‘Stimulus Decoding’), was successively 
decoded and transferred between Rats 2 and 3, and again transferred to Rat 1, across a period of four trials 
(memory trace in red). The use of the brain-to-brain interface between the nodes of the network allowed 
accurate transfer of information. B) The overall performance of the Brainet was significantly better than 
the performance in the No-ICMS sessions and better than individual rats performing 4 consecutive correct 
trials. In this panel, * indicates P< 0.05 and *** indicates P< 0.001. C) Neuron dropping curve of Brainet 
for storage and retrieval of memories. D) Example of session with multiple memories (each column) 
processed in blocks of four trials (each row). Information flows from the bottom (Stimulus delivered) 
towards the top (Trials 1-4). Blue and red indicate Stimulus 1 or 2 respectively. Correct tactile memory 
traces are columns which have a full sequence of trials with the same color (see blocks: 3, 5, 7 and 9). In this 
panel, * indicates an incorrect trial. 


progressing to the top one. The experiment started by delivering one of two different ICMS stimuli to the 
S1 of the input rat (from now on referred to as Rat 1) during the first trial (Trial 1). Neuronal ensemble 
activity sampled from Rat 1 was then used to decode the identity of the stimulus (either Stimulus 1 or 
2). Once the stimulus identity was determined, a new trial started and a BtBI was employed to deliver 
a correspondent ICMS pattern to Rat 2, defining Trial 2 of the task. In this arrangement, the BtB link 
between Rat 1 and Rat 2 served to store the pattern (Pattern Storage I). Next, neuronal ensemble activity 
was recorded from the S1 of Rat 2. In the third trial, it was Rat 3’s turn to receive the tactile message 
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(Pattern Storage II) decoded from the neural ensemble activity of Rat 2, via an ICMS mediated BtB link. 
During the fourth and final trial, Rat 1 received the message decoded from the neural activity of Rat 3. 

Using this Brainet architecture, the memory of a tactile stimulus could only be recovered if the indi- 
vidual BtB communication links worked correctly in all four consecutive trials. The chance level for this 
operation was 6.25%. Under these conditions, this Brainet was able to retrieve a total of 35.37 + 2.2% 
(9 sessions in 9 rats) of the tactile stimuli presented to it (Kruskall Wallis statistic = 14.89; P= 0.0006, 
Fig. 5B), contrasting with 7.91 + 6.5% in No-ICMS sessions (n=5 sessions; Dunn's test: P < 0.001). For 
comparison purposes, individual rats performed the same four-trial task correctly in only 15.63 + 2.1% 
of the trials. This outcome was significantly lower than a 3-rat Brainet (Dunn's test: P< 0.001). As in the 
previous experiments, larger neuronal ensembles yielded better encoding (Fig. 5C). 

As an additional control, rats that were not processing memory related information in a specific trial 
(e.g. Rats 2 and 3 during the Stimulus Decoding Stage in Rat 1) received Stimulus 1 or Stimulus 2, ran- 
domly chosen. Thus, in every single trial all rats received some form of ICMS, but only the information 
gathered from a specific rat was used for the overall tactile trace. 

The colored matrix in Fig. 5D illustrates a session in which a tactile trace developed along the 3-rat 
Brainet. A successful example of information transfer and recovery is shown in the third block of trials 
(blue column on the left). The figure shows that the original stimulus (Stimulus 1 - bottom blue square) 
was delivered to the S1 of Rat 1 in the first trial. This stimulus was successfully decoded from Rat 1’s 
neural activity, as shown by the presence of the blue square immediately above it (Trial 1 - Stimulus 
Decoding). In Trial 2 (Pattern Storage I), Stimulus 2 was delivered, via ICMS to the S1 of Rat 2, and again 
successfully decoded (as shown by the blue square in the center). Then, in Trial 3 (Pattern Storage II), 
the ICMS pattern delivered to Rat 3 corresponded to Stimulus 1, and the decoding of $1 neural activity 
obtained from this animal still corresponded to Stimulus 1, as shown by the blue square. Lastly, in Trial 
4 (Stimulus Recovery), Rat 1 received an ICMS pattern corresponding to Stimulus 1 and its S1 neural 
activity still encoded Stimulus 1 (blue square). Thus, in this specific block of trials, the original tactile 
stimulus was fully recovered since all rats were able to accurately encode and decode the ICMS pattern 
received. Similarly, columns 5, 7, and 9 also show blocks of trials where the original tactile stimulus (in 
these cases Stimulus 2, red square) was accurately encoded and decoded by the Brainet. Alternatively, 
columns with an asterisk on top (e.g. 1 and 8) indicate incorrect blocks of trials. In these incorrect blocks, 
the stimulus delivered was not accurately encoded in the brain of at least one rat belonging to the Brainet 
(e.g. rat 3 in block 1). 


Brainet for sequential and parallel processing. Lastly, we combined all the processing abilities 
demonstrated in the previous experiments (discrete tactile stimulus classification, BtB interface, and 
tactile memory storage) to investigate whether Brainets would be able to use sequential and parallel 
processing to perform a tactile discrimination task (N=5 rats in N=10 sessions). For this we used 
blocks of two trials where tactile stimuli were processed according to Boolean logic'® (Fig.6A-B). This 
means that in each trial there was a binary decision tree (i.e. two options encoded as Stimulus 1 or 2). 
In the first trial, two different tactile inputs were independently sent to two dyads of rats (Dyad 1: Rat 
1-Rat 2; Dyad 2: Rat 3-Rat 4; bottom of Fig. 6A). In the next trial, the tactile stimuli decoded by the 
two dyads were combined and transmitted, as a new tactile input, to a 4-rat Brainet. Upon receiving this 
new stimulus, the Brainet was in charge of encoding a final solution (i.e. identifying Stimulus 3 or 4, see 
Supplementary Figure 2). 

As shown at the bottom of Fig. 6A, odd trials were used for parallel processing, ie. each of two rat 
dyads independently received ICMS patterns, while neural activity was analyzed and the original tactile 
stimulus decoded (i.e. Stimulus 1 or 2). Then, during even trials (Fig. 6A, top), ICMS was used to encode 
a second layer of patterns, defined as Stimulus 3 and Stimulus 4. Note that ICMS Stimuli 3 and 4 were 
physically identical to Stimuli 2 and 1 respectively; however, because the stimuli delivered in the even 
trials were contingent on the results of the odd trials, we employed a different nomenclature to identify 
them. The decision tree (i.e. truth table) used to calculate the stimuli for the even trials is shown in the 
colored matrix at the center of Fig. 6A. The matrix shows that, if both dyads encoded the same tactile 
stimulus in the odd trial (Stimulus 1-Stimulus 1, or Stimulus 2-Stimulus 2; combinations with blue encas- 
ing), the ICMS delivered to the entire Brainet in the even trial corresponded to Stimulus 4. Otherwise, if 
the tactile stimulus decoded from each rat dyad in the odd trial was different (Stimulus 1-Stimulus 2, or 
Stimulus 2-Stimulus 1; combinations with red encasing), the ICMS delivered to the entire Brainet in the 
even trial corresponded to Stimulus 3. As such, the ICMS pattern delivered in even trials was the same 
for the whole Brainet (i.e. all four rats). 

At the end of each even trial, the stimulus decoded from the combined neuronal activity of the four 
brain ensemble (top of Fig. 6A) defined the final output of the Brainet. Chance level was set at 12.5%. 
Overall, this Brainet performance was much higher than chance level or No-ICMS sessions (Brainet: 
45.22+ 3.4%, n=10 sessions) significantly above No-ICMS sessions (n=5 sessions) (No-ICMS: 
22.79 + 5.4%; Kruskal-Wallis statistic = 7.565, P = 0.0228; Dunn's test: P < 0.05 Fig. 6C). Additionally, the 
Brainet also outperformed each individual rat (groups of three consecutive trials: 30.25 + 3.0%; Dunn’s 
test: P< 0.05). 

As our last experiment, we tested whether a 3-rat Brainet could be used to classify meteorological 
data (see Methods for details). Again, the decision tree included two independent variables in the odd 
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Figure 6. A Brainet for parallel and sequential processing. A) Architecture of a network for Parallel and 
Sequential processing. Information flows from the bottom to the top during the course of two trials. In first 
trial, odd trial for parallel processing, Dyad 1 (Rat 1-Rat 2) received one of two ICMS patterns, and Dyad 

2 (Rat 3-Rat 4) received independently one of two ICMS patterns. During Trial 2, even trial for sequential 
processing, the whole Brainet received again one of two ICMS patterns. However, the pattern delivered in 
the even trial was dependent on the results of the first trial and was calculated according to the colored 
matrix presented. As depicted by the different encasing of the matrix (blue or red), if both dyads encoded 
the same stimulus in the odd trial (Stimulus 1-Stimulus1 or Stimulus 2-Stimulus 2), then the stimulus 
delivered in the even trial corresponded to Stimulus 3. Otherwise, if each dyad encoded a different stimulus 
in the odd trial (Stimulus1-Stimulus 2 or Stimulus 2-Stimulus 1), then the stimulus delivered in even trial 
was Stimulus 4. Each correct block of information required three accurate estimates of the stimulus delivered 
(ie. encoding by both dyads in the even trial, as well as the whole Brainet in the odd trial). B) Example of 
session with sequential and parallel processing. The bottom and center panel show the dyads processing the 
stimuli during the odd trials (parallel processing), while the top panel shows the performance of the whole 
Brainet during the even trials. In this panel, * indicates an incorrect classification. C) The performance 

of the Brainet was significantly better than the performance during the No-ICMS sessions and above the 
performance of individual rats performing blocks of 3 correct trials. In this panel, * indicates P< 0.05. 


trials and a dependent variable in the even trials (see Supplementary Figure 3). Figure 7A illustrates how 
Boolean logic was applied to convert data from an original weather forecast model . In the bottom panel, 
the yellow line depicts continuous changes in temperature occurring during a period of 10 hours. Periods 
where the temperature increased were transferred to the Brainet as Stimulus 1 (see arrows in periods 
between 0 and 4hours), whereas periods where the temperature decreased were transferred as Stimulus 
2 (see arrows in periods between 6 and 10hours). The middle panel of Fig. 7A illustrates changes in 
barometric pressure (green line). Again, periods where the barometric pressure increased were translated 
as Stimulus 1 (e.g. between 1-2hours), while periods where the barometric pressure decreased were 
translated as Stimulus 2 (e.g. 3-5 hours). 

Both Stimulus 1 and 2 were delivered to a Brainet during odd trials; changes in temperature were 
delivered to Rat 1 alone, while changes in barometric pressure were delivered to Rats 2 and 3. As in 
the previous experiment, Stimuli 3 and 4 were physically similar to Stimuli 1 and 2. In even trials, 
increases and decreases in the probability of precipitation (top panel Fig. 7A) were calculated as fol- 
lows: an increase in temperature (Stimulus 1; Rat 1) combined with a decrease in barometric pressure 
(Stimulus 2; Rats 2 and 3) was transferred to even trials as an increase in the probability of precipitation 
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Figure 7. Parallel and sequential processing for weather forecast A) Each panel represents examples of 
the original data, reflecting changes in temperature (lower panel), barometric pressure (center panel), and 
probability of precipitation (upper panel). The arrows represent general changes in each variable, indicating 
an increase or a decrease. On the top of each panel is represented the ICMS pattern that resulted from 
each arrow presented. B) Lower and center panels show trials where different rats of the Brainet (Rat 1 
lower panel, and Rats 2-3 center panel) processed the original data in parallel. Specifically, Rat 1 processed 
temperature changes and Rats 2-3 processed barometric pressure changes. The upper panel shows the 
Brainet processing changes in the probability of precipitation (Rats 1-3) during the even trials. * indicates 
trials where processing was incorrect. 


(i.e. a Stimulus 4), whereas any other combination was transferred as Stimulus 3, and associated with a 
decrease in precipitation probability. This specific combination of inputs was used because it reflects a 
common set of conditions associated with early evening spring thunderstorms in North Carolina. 

Overall, our 3-rat Brainet predicted changes in the probability of precipitation with 41.02 +5.1% 
accuracy which was much higher than chance (No-ICMS: 16.67 + 8.82%; n= 3 sessions; t= 2.388, df= 4; 
P=0.0377) (also see Fig. 7B). 


Discussion 
In this study we described different Brainet architectures capable of extracting information from multiple 
(3-4) rat brains. Our Brainets employed ICMS based BtBs combined with neuronal ensemble recordings 
to simultaneously deliver and retrieve information to and from multiple brains. Multiple BtBIs were used 
to construct some of our Brainet designs. Our experiments demonstrated that several Brainet architec- 
tures can be employed to solve basic computational problems. Moreover, in all cases analyzed the Brainet 
performance was equal or superior to that of an individual brain. These results provide a proof of concept 
for the possibility of creating computational engines composed of multiple interconnected animal brains. 
Previously, Brainets have incorporated only up to two subjects exchanging motor or sensory infor- 
mation’, or up to three monkeys that collectively controlled the 3D movements of a virtual arm*. These 
studies provided two major building blocks for Brainet design: (1) information transfer between indi- 
vidual brains, and (2) collaborative performance among multiple animal brains. Here, we took advan- 
tage of these building blocks to demonstrate more advanced Brainet processing by solving multiple 
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computational problems, which included discrete classification, image processing, storage and retrieval 
of memories, and a simplified form of weather forecasting’”*. All these computations were dependent on 
the collective work of cortical neuronal ensembles recorded simultaneously from multiple animal brains 
working towards a common goal. 

One could argue that the Brainet operations demonstrated here could result from local responses of 
S1 neurons to ICMS. Several lines of evidence suggest that this was not the case. First, we have demon- 
strated that animals needed several sessions of training before they learned to synchronize their S1 activ- 
ity with other rats. Second, the decoding for individual neurons in untrained rats was close to chance 
levels. Third, attempts to make the Brainet work in anesthetized animals resulted in poor performance. 
Fourth, network synchronization and individual neuron decoding failed when animals did not attend 
to the task requirements and engaged in grooming instead. Fifth, removing the reward contingency 
drastically reduced the Brainet performance. Sixth, after we reduced trial duration, the decoding from 
individual neurons dropped to levels close to chance. 

Altogether, these findings indicate that optimal Brainet processing was only attainable in fully awake, 
actively engaged animals, with an expectation to be rewarded for correct performance. These features 
are of utmost importance since they allowed Brainets to retain the computational aptitudes of the awake 
brain'! and, in addition, to benefit from emergent properties resulting from the interactions between 
multiple individuals’. It is also noteworthy to state that the Brainets implemented here only allowed par- 
tial social interactions between subjects (through the Plexiglas panels). As such, it is not clear from our 
current study, to what extent social interactions played (or not) a pivotal role in the Brainet performance. 
Therefore, it will be interesting to repeat and expand these experiments by allowing full social contact 
between multiple animals engaged in a Brainet operation. In this context, Brainets may become a very 
useful tool to investigate the neurophysiological basis of animal social interactions and group behavior. 

We have previously proposed that the accuracy of the BtBI could be improved by increasing the 
number of nodes in the network and the size of neuronal ensembles utilized to process and transfer 
information’. The novel Brainet architectures tested in the present study support these suggestions, as 
we have demonstrated an overall improvement in BtBI performances compared to our previous study 
(maximum of 72% correct in the previous study versus maximum of 87% correct here)’. Since neuron 
dropping curves did not reach a plateau, it is likely that the performance of our Brainet architectures 
can be significantly improved by the utilization of larger cortical neuronal samples. In addition, switch- 
ing between sequential and parallel processing modes, as was done in the last experiment, allowed the 
same Brainet to process more than two bits of information. It is important to emphasize, however, that 
the computational tasks examined in this study were implemented through Boolean logic’®!”. In future 
studies we propose to address a new range of computational problems by using simultaneous analog and 
digital processing. By doing so, we intend to identify computational problems that are more suitable for 
Brainets to solve. Our hypothesis is that, instead of typical computational problems addressed by digital 
machines, Brainets will be much more amenable to solving the kind of problems faced by animals in 
their natural environments. 

The present study has also shown that the use of multiple interconnected brains improved Brainet 
performance by introducing redundancy in the overall processing of the inputs and allowing groups 
of animals to share the attentional load during the task, as previously reported for monkey Brainets®. 
Therefore, our findings extended the concept of BtBIs by showing that these interfaces can allow net- 
works of brains to alternate between sequential and parallel processing!’ and to store information. 

In conclusion, we propose that animal Brainets have significant potential both as a new experimen- 
tal tool to further investigate system neurophysiological mechanisms of social interactions and group 
behavior, as well as provide a test bed for building organic computing devices that can take advantage of 
a hybrid digital-analogue architecture. 


Methods 

All animal procedures were performed in accordance with the National Research Council’s Guide for the 
Care and Use of Laboratory Animals and were approved by the Duke University Institutional Animal 
Care and Use Committee. Long Evans rats weighing between 250-350 g were used in all experiments. 


Tasks of synchronization and desynchronization. Groups of four rats, divided in two pairs 
(dyads), were placed in two behavioral chambers (one dyad in each chamber). Rats belonging to the same 
dyad (i.e. inside the same chamber) could see each other through a Plexiglas panel, but not the animals 
in the other dyad. Each trial in a session consisted of four different periods: baseline (from 0-9 seconds), 
ICMS (9-11 seconds), test (11-12 seconds), and reward (13-25 seconds). During the baseline period no 
action was required from rats. During the ICMS period a pattern of ICMS (20pulses, at 22-26 Hz, 
10-100uA) was delivered to all rats simultaneously. During the Test period, neural activity from all 
neurons recorded in each rat was analyzed and compared to the neural activity of all other animals as a 
population. Spikes from individual channels were summed to generate a population vector representing 
the overall activity which generally constitutes a good indicator of whisking and/or licking activity'*. 
The population vectors for each of the four rats were then normalized. Lastly, we calculated the Pearson 
correlation between the normalized population vector of each rat and the general population of rats (the 
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average of the neural population vectors from three remaining rats). During Pre-Sessions neural activity 
was analyzed in each trial, but no ICMS or water reward was delivered. During Sessions, neural activity 
was analyzed after the delivery of an ICMS stimulus and if the threshold for a correct trial was reached 
(at least three rats with R> =0.2) then a water reward was delivered. During the Post-Sessions, neural 
activity was recorded and a water reward was delivered if animals reached the threshold for a correct 
trial, however no ICMS stimuli were delivered. 

Additionally, we also tested the effect of ICMS alone and in anesthetized animals (Ketamine/Xylazine 
100 mg/kg). During the synchronization/desynchronization task two different ICMS patterns were deliv- 
ered: Stimulus 1 consisted of the same pattern that was used for the synchronization task and the thresh- 
old for a correct trial remained the same. Stimulus 2 consisted of two short bursts of ICMS (2 x 4pulses, 
22-26 Hz separated by 250 ms interval) and the threshold for a correct response was less than three rats 
reaching an R value of 0.2 during the testing period. 


Adaptive decoding algorithm. During the experiments where the adaptive decoding algorithm was 
used (discrete classification, tactile memory storage, sequential and parallel processing), the ICMS pat- 
terns remained as previously. Neural activity was separately analyzed for each neuron in each rat and 
25 ms distributions were built and filtered with a moving average of 250 ms. The overall structure of the 
sessions included an initial period of 16-30 trials where Stimuli 1 and 2 were delivered to rats in order 
to build the distributions for each stimulus. The overall firing rate for each bin in the test period was 
then analyzed and, according to the probability distributions, a vote for Stimulus 1 or for Stimulus 2 was 
calculated. Bins with similar spike distributions for both stimuli were not analyzed. A final vote for each 
cell was then calculated, using the votes from all the bins that presented differences in the firing rate for 
the two stimuli. Lastly, the final votes for each cell in the population were filtered with a sigmoid curve. 
This filtering allowed the best encoding cells in the ensembles to contribute significantly more than other 
cells to the overall decision made by the Brainet made in each trial. Additionally, the weight of the cell 
population could be automatically adjusted at different intervals (e.g. every 10 or 15 trials). 

For the image processing experiment, groups of four rats were tested. An original image was pixilated 
and converted into multiple trials. Each trial corresponded to a white (Stimulus 1) or blue (Stimulus 2) 
pixel in the original image. In each trial one of two different ICMS stimuli was delivered to the Brainet. 
After the neural activity from the Brainet was decoded, a new image corresponding to the overall pro- 
cessing by the Brainet was recreated. 


Memory storage experiment. For this specific experiment only three rats were used in each session 
and ICMS frequency patterns varied between 20-100 Hz. The number of pulses remained the same as in 
the previous experiments. Each memory was processed across a period of four trials which represented 
four different stages of a memory being processed: Stimulus delivery (Trial 1), Pattern Storage I (Trial 2), 
Pattern Storage II (Trial 3), and lastly, Stimulus Recovery (Trial 4). Information was initially delivered to 
the S1 cortex of the first rat (Rat 1) in the first trial - Stimulus Delivery. In Trial 2, information decoded 
from the cortex of Rat 1 was delivered as an ICMS pattern to the second rat (Rat 2) - Pattern Storage I. 
In Trial 3, information decoded from the S1 of Rat 2 was delivered to Rat 3 - Pattern Storage II. In Trial 
4, neural activity decoded from the cortex of Rat 3 was decoded and delivered to the cortex of Rat 1 as a 
pattern of ICMS. Lastly, if the stimulus encoding and decoding was correct across all four trials (chance 
level of 6.25%) a memory was considered to be recovered. The overall number of memories decoded, the 
percent of stimuli decoded and the accuracy of the brain-to-brain interface information transfer were 
measured. As a control measure the Plexiglas panels separating the dyads were made opaque for this 
experiment. Additionally, as the tactile pattern was delivered to each rat in the specific memory stage 
(delivery, storage or recovery), a random Stimulus 1 or 2 was delivered to the remaining rats. This ran- 
dom stimulation of the remaining individuals ensured that, in each trial, rats could not identify whether 
or not they were participating in the tactile trace. 


Sequential and parallel processing experiment. Each block of information processing consisted 
of two trials: the first trial corresponded to parallel processing and the second trial corresponded to 
sequential processing. Two dyads of rats were formed: Dyad 1 (Rat 1-Rat 2) and Dyad 2 (Rat 3-Rat 4). 
During the first trial each dyad processed one of two ICMS stimuli independently of the other dyad. 
After the delivery of the ICMS stimuli to each dyad, neural activity was decoded and the stimulus for 
Trial 2 was computed from the results. If both dyads encoded a similar stimulus (Stimulus 1 - Stimulus 
1, or Stimulus 2 - Stimulus 2), then the ICMS stimulus in Trial 2 was Stimulus 3. Otherwise, if the dyads 
encoded different ICMS stimuli (Stimulus 1 - Stimulus 2, or Stimulus 2 - Stimulus 1), then the ICMS 
stimulus in Trial 2 would be Stimulus 4. Stimuli 1 and 3 and Stimuli 2 and 4 had the exact same physical 
characteristics (number of pulses). During the second trial the same stimulus was delivered simultane- 
ously to all four rats, and the Brainet encoded an overall response. A block of information was considered 
to be correct only if both Trials 1 and 2 were correct in both the dyads and in the Brainet. 

For the weather forecasting experiment groups of three animals were tested. Sessions were run as 
described above for sequential and parallel processing. However, Trial one (parallel processing) was 
processed only by one rat (temperature) and one dyad of rats (barometric pressure), while Trial two 
(sequential processing: probability of precipitation) was processed by the whole Brainet (three rats). 
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To establish a simple weather forecast model we used original data from Raleigh/Durham Airport 
(KRDU), at WWW.Wunderground.com. Estimates were collected on August 2, 2014. We used periods 
characterized by increases and decreases in temperature and barometric pressure as independent varia- 
bles, and increases in the probability of precipitation as the dependent variable. A total of 13 periods were 
collected. These included a total of 26 independent inputs for even trials (13 variations in temperatures 
and 13 variations in barometric pressure), as well as 13 additional changes in the probability of precipi- 
tation, to be compared with the Brainet outputs (i.e. the actual forecast). Specifically, for this experiment, 
increases in temperature (Stimulus 1 for the first rat) with decreases in barometric pressure (Stimulus 
2 in Rats 2-3), during the odd trials, were computed as an increase in the probability of precipitation 
(Stimulus 4 to the Brainet in the even trial). Otherwise, increases or decreases in temperature (Stimulus 
1 or 2 in the odd trial) combined with an increase in barometric pressure (Stimulus 1 for Rats 2 and 3), 
were computed as a decrease in the probability of precipitation (Stimulus 3 for the Brainet) in the even 
trial. Stimuli 1 and 3, and Stimuli 2 and 4 had the exact same physical characteristics (number of pulses). 


Surgery for microelectrode array implantation. Fixed or movable microelectrode bundles or 
arrays of electrodes were implanted bilaterally in the S1 of rats. Craniotomies were made and arrays 
lowered at the following stereotaxic coordinates: [(AP) —3.5mm, (ML), £5.5mm (DV) —1.5mm]. 


Electrophysiological recordings. A Multineuronal Acquisition Processor (64 channels, Plexon Inc, 
Dallas, TX) was used to record neuronal spikes, as previously described’. Briefly, differentiated neural 
signals were amplified (20000-32,000 x) and digitized at 40 kHz. Up to four single neurons per recording 
channel were sorted online (Sort client 2002, Plexon inc, Dallas, TX). 


Intracortical electrical microstimulation. Intracortical electrical microstimulation cues were gen- 
erated by an electrical microstimulator (Master 8 , AMPI, Jerusalem, Israel) controlled by custom Matlab 
script (Nattick, USA) receiving information from a Plexon system over the internet. Patterns of 8-20 
(bipolar, biphasic, charge balanced; 200,1sec) pulses at 20-120 Hz were delivered to $1. Current intensity 
varied from 10-100 1A. 
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While it is well known that the primate brain evolved to cope with complex social contingencies, the 
neurophysiological manifestation of social interactions in primates is not well understood. Here, 
concurrent wireless neuronal ensemble recordings from pairs of monkeys were conducted to measure 
interbrain cortical synchronization (ICS) during a whole-body navigation task that involved continuous 
social interaction of two monkeys. One monkey, the passenger, was carried in a robotic wheelchair to 
a food dispenser, while a second monkey, the observer, remained stationary, watching the passenger. 
The two monkeys alternated the passenger and the observer roles. Concurrent neuronal ensemble 
recordings from the monkeys’ motor cortex and the premotor dorsal area revealed episodic occurrence 
of ICS with probability that depended on the wheelchair kinematics, the passenger-observer distance, 
and the passenger-food distance - the social-interaction factors previously described in behavioral 
studies. These results suggest that ICS represents specific aspects of primate social interactions. 


Observing the behaviors of others is essential for primates, including humans, to be able to handle the complex 
dynamics of their social groups. Such observations may allow individuals to learn social ranks, recognize threats 
and potential allies, as well as learn new motor skills’. Experiments in monkeys have demonstrated that, while an 
animal observes the actions performed by a different subject, frontal and parietal neurons of the observer respond 
as if the observer performed the same action by itself*-'*. Such cortical neurons, which represent the actions of 
others, are classically known as “mirror neurons”»». 
Studies of mirror neurons have provided insights on how cortical neuronal ensembles mediate social inter- 
actions through imitation and motor cooperation'*~*. Several of these studies employed interactive tasks where 
monkey pairs cooperated or competed'*”**. Such experiments usually employed single-unit recordings from 
one of the subjects’ brains, but not large-scale neuronal ensemble recordings obtained simultaneously from both 
brains of the interacting animals. This shortcoming resulted mostly from the major technical challenges involved 
in obtaining concurrent neuronal recordings from multiple subjects. Because of this limitation, the neuronal 
correlates of social interaction among multiple subjects have not been fully investigated in studies dealing with 
mirror neuron activity. 
In the present study, we overcame this technical barrier by introducing a new paradigm that allowed mul- 
tichannel wireless recording from the brains of a monkey pair engaged in a whole-body navigation task. Using 
: this new neurophysiological approach, we asked how cortical activity recorded simultaneously from two mon- 
: keys could reflect such parameters as the animals’ relative position in space and their closeness to food - crucial 
: factors determining primate social interaction**”*. In the whole-body navigation paradigm, pairs of monkeys 

alternatively played one of two distinct roles from day to day: while one monkey, the passenger, was carried 
: by a motorized wheelchair, another, the observer, was seated in a stationary chair, watching the passenger’s 
: whole-body movement. Both monkeys were motivated to attend to the wheelchair movements because they were 
: both rewarded upon the passenger reaching the target: the passenger collected grapes, while the observer received 
: juice. Our analysis showed that ICS between the two monkeys reflected the passenger’s whole-body movements 
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Figure 1. Interbrain cortical synchronization (ICS) during the navigation task. (A) Locations of cortical 
implants in three monkeys (C,J, and K). Neuronal-ensemble recordings were conducted in M1 (red dots) and 
PMd (blue dots), in both hemispheres. (B) The experimental setup. Two monkeys (passenger and observer) 
were placed in a 5.0-by-3.9 m room. The passenger sat in an electrically actuated wheelchair. The observer sat 
in a stationary chair placed in the corner of the room. During each trial, the passenger moved from a starting 
location (shown on the left) to a stationary grape dispenser. Five representative routes of the wheelchair 

are plotted in different colors. These routes were randomly generated by a computer program. (C) Color 

plots of neuronal-ensemble activity for two representative trials. Each horizontal line corresponds to a unit. 
Color represents normalized (z-scored) firing rate of 69 units were recorded in monkey C (observer in 

this experiment) and 47 in monkey K (passenger). Episodes of ICS are marked by red horizontal lines. (D) 
Continuous evaluation of ICS for the trials shown in (C). Instantaneous values of the distance correlation were 
computed with a sliding window, of the same 3-s width as the red bars in (C). Correlation peaks are marked by 
arrows. (E) Wheelchair routes for the same trials as in (C) and (D). The routes are color-coded to indicate ICS. 


and spatial location, the distance between the passenger and observer, and the distance between the passenger 
and food reward. 


Results 

Experiments were conducted in three female monkeys, monkeys C, J and K. Of these, monkey C was the most 
dominant, as judged by the feeding priority in conflict over food”**', followed by monkey K, and then by mon- 
key J (see Methods: Social ranking). Monkeys were chronically implanted with multiple cortical multielec- 
trode arrays. A monkey pair participated in each experimental session; monkey pairs C-K and C-J were tested. 
Neuronal ensemble activity was recorded in both monkeys simultaneously, using a 256-channel wireless record- 
ing system****. In monkey J, we recorded bilaterally from the primary motor cortex (M1), and in monkeys C 
and K, bilaterally from M1 and dorsal premotor cortex (PMd) (Fig. 1A). The number of recorded units ranged, 
depending on the session, 66-68 in monkey J, 70-90 in monkey C (M1, 43-54; PMd, 27-36), and 43-48 in mon- 
key K (M1, 30-33; PMd, 43-48). 

During the experiments, animals were seated in their monkey chairs and placed inside a 5.0m-by-3.9m room 
(Fig. 1B). The chair of the monkey, called observer, remained stationary. The chair of the other monkey, called 
passenger, was mounted on an electrically actuated cart that traveled freely in the room. The task consisted of the 
passenger navigating toward a grape dispenser to collect a grape. At the time the passenger obtained the grape, 
the observer also received a juice reward. The grape dispenser was situated in the corner of the room; the observer 
stayed in another corner. The passenger started navigation from a location near the wall opposing the grape 
dispenser and observer. For both monkey pairs (C-K and C-J), the passenger and observer roles were alternated 
among monkeys in different experiments. The wheelchair was moved by a computer program along a randomly 
generated trajectory. 


Interbrain cortical synchronization between the passenger and observer. We observed that 
cortical activity concurrently recorded in both the passenger and observer exhibited episodic ICS, where the 
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Figure 2. The dependence of ICS on wheelchair position and velocity, expressed as conditional probability of 
ICS episodes. (A-D) Probability of ICS episodes as a function of room x,y coordinates. The values of probability 
at each pixel are averaged with immediate neighbors and are color coded. The one-dimensional bars below 

each two-dimensional plot show the dependence of probability on the distance between the passenger and food 
reward (top), and between the passenger and the observer (bottom). The results for monkey pair C-K are shown 
in (A,C) and for monkey pair C-J in (B,D). Monkey C was either passenger (panels on the left) or observer 
(panels on the right). In (A,B) the observer was in the right corner of the room (relative to the passenger at 

the starting location), and the grape dispenser was in the left corner. In (C,D), the locations of the observer 

and grape dispenser were swapped. (E,F) Color-coded probability of ICS episodes as a function of wheelchair 
velocity for monkey pairs C-K (E) and C-J (F). Horizontal axis corresponds to rotational velocity, and vertical 
axis corresponds to translational velocity. (See Fig. $4 for the number of samples contributed to each pixel). 


correlation of neuronal firing patterns between the two monkeys was significantly higher than the one obtained 
for permuted data (i.e., null distribution). While the cause of this ICS was not apparent in individual trials, sta- 
tistical analysis of the ICS episodes (see Methods: Episodic ICS) showed that their probability of occurrence 
depended on wheelchair position and velocity, the distances between the passenger and grape dispensers, and the 
distance between the passenger and observer. Figure 1C-E shows two representative trials containing several ICS 
episodes, with correlation coefficients as high as 0.5. ICS episodes constituted 19.7% + 2.1% (mean + standard 
error) of the total session time for monkey pair C-K, and 35.7% + 4.7% for C-J (Supplementary Fig. $1). 

To determine the factors influencing such an ICS, we calculated the probability of ICS episodes as a function 
of different parameters of the wheelchair movements: translational and rotational velocity, room coordinates 
of the wheelchair, distance from the wheelchair to the grape dispenser, and distance from the wheelchair to the 
observer (see Methods: ICS probability). We found that ICS probability depended on each of these parameters, 
and was also influenced by monkey pair composition and task roles (passenger vs. observer) assigned to each 
monkey. 

Figure 2 illustrates how ICS probability depended on the wheelchair position (Fig. 2A-D) and velocity 
(Fig. 2E,F). When monkey C was paired with monkey K, and C was the passenger, episodes of high ICS became 
more frequent when the distance between the monkeys decreased (Fig. 2A,C, left panels, where the two distri- 
butions look like mirrored images as the observer and the reward swapped locations). This relationship reversed 
when monkey C was the observer (Fig. 2A,C, right panels): synchrony episodes occurred less frequently when 
the monkeys were close to each other. For the same monkey pair, the probability of ICS episodes also depended 
on the wheelchair velocity. This dependence also reversed after the monkeys’ roles changed (compare left and 
right panels in Fig. 2E). Similar effects were observed for monkey pair C-J. Wheelchair position, velocity and 
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monkey roles influenced the probability of ICS (Fig. 2B,D,F). Notably for this monkey pair, higher levels of ICS 
were frequent when monkey C, as the passenger, was far away from monkey J (Fig. 2D, left), the observer. When 
C approached J, ICS episodes became less frequent. The dependence of ICS on the passenger’s position changed 
dramatically after C became the observer (Fig. 2D, right), as did the dependence of ICS on wheelchair velocity 
(Fig. 2F). 

In addition to the effect of the distance between the monkeys, ICS probability was affected by the distance 
between the passenger and the grape dispenser (Fig. 2A-D). These ICS patterns depended on the monkey pair 
composition. For example, in monkey pair C-K, the probability of ICS episodes increased when the passen- 
ger (C or K) approached the grape dispenser (Fig. 2A,C; Spearman correlation = —0.28 + 0.18). Conversely, for 
pair C-J, the ICS decreased when the passenger (C or J) was close to the grape dispenser (Fig. 2B,D; Spearman 
correlation = 0.47 £0.17). 

While ICS probability clearly depended on where the passenger was positioned relative to the observer and 
the grape dispenser, these neural patterns could also reflect room landmarks, for example, left versus right corner. 
We tested for this possibility by swapping the locations of the grape dispenser and observer on different sessions. 
This manipulation produced a reversal of the ICS patterns (e.g., compare Fig. 2A and C), indicating that room 
landmarks were not as important as the positions of the observer and the grape dispenser. 

ICS probability also depended on the wheelchair velocity. In the example of Fig. 2E, the dominant monkey C 
was the passenger and monkey K was the observer. In these settings, ICS probability increased when the wheel- 
chair had high rotational velocity. However, when the monkeys’ roles were swapped, the ICS probability did not 
vary with increases of the wheelchair rotational velocity. ICS patterns depended on the wheelchair velocity in the 
monkey pair C-J, as well (Fig. 2F). 

Since in the monkey pair C-K we recorded from M1 and PMd in both animals, we could compare the engage- 
ment of each of these cortical areas in ICS. For every ICS episode, determined by the analysis for the entire neu- 
ronal sample, we measured whether a particular pair of cortical areas exhibited ICS (permutation test, p < 0.05). 
When monkey K was the observer, M1-to-M1 synchronization was found to be most frequent (60.5% + 1.3% 
of the episodes), followed by M1-to-PMd (51.5% + 1.0%) and PMd-to-PMd (42.5% + 1.4%). The same trend 
was found when monkey C was the observer: 53.0% + 1.3% for M1-to-M1, 47.0% + 1.0% for M1-to-PMd, and 
43.6% + 1.4% for PMd-to-PMd. 


Neuronal modulation to wheelchair velocity and acceleration. Having established that ICS 
depended on the passenger’s position and velocity, we examined the relationship between the presence of ICS 
and the modulation of neuronal firing rates to position and velocity, in both the passenger’s and observer's cortex. 
For that, we first evaluated neuronal modulation patterns to wheelchair kinematics, i.e. velocity and acceleration 
(Fig. 3). This analysis revealed that the passenger had more velocity and acceleration modulated units in both M1 
and PMd (x?(1) > 9.35, p < 0.05). In the passenger’s PMd, 40.6% and 19.5% units (data from all monkey pairs 
combined) were modulated to velocity and acceleration, respectively, whereas in the observer’s PMd these values 
dropped to 0.8% and 0%. Considering M1, the corresponding values were 43.8% and 15.5% for the passenger 
and 0.5% and 0% for the observer. The dependence of neuronal firing rate on different parameters of naviga- 
tion was quantified using modulation depth as a metric (see Methods: Modulation depth). Modulation depth 
for both velocity and acceleration was on average higher in the passenger compared to the observer (ANOVA, 
F(1,1434) > 270.2 for monkey role, p < 0.05; See Table $1, S2 and Methods: ANOVA for modulation depth). 
Additionally, significant differences were found when M1 modulation depth was compared to that of PMd (per- 
mutation test, p< 0.05). Monkey pair-dependent effects were found: in monkey C, modulation was stronger 
in PMd than M1 when C was paired with K but weaker when C was paired with J (permutation test, p < 0.05) 
(Table S9). 

To assess whether there was any correspondence between the modulation depth values during navigating 
and observing, we calculated the correlation coefficient between the neuronal population features recorded in 
different areas (Fig. S2). We found significant correlations for both M1 (Spearman correlation rho = 0.24-0.64, 
p<0.05) and PMd (rho =0.48-0.81, p <0.05). 

The presence or absence of ICS clearly correlated with changes in modulation of neuronal rates to wheel- 
chair kinematics (Fig. 3A,B, top vs. bottom). Figure 3A,B shows two representative units. The first was recorded 
in monkey C’s PMd when C was paired with K (Fig. 3A). When monkey C was the passenger, this PMd unit 
was non-directionally modulated to rotational velocity: its firing rate increased for both rightward and leftward 
rotations (see X-axis). Yet, during ICS episodes in which monkey C became the observer, this PMd unit’s veloc- 
ity modulation pattern became more prominent since its neuronal firing rate increased for rightward rotations 
and decreased for leftward rotations (Fig. 3A, right, see X-axis). The second example illustrated is an M1 unit 
recorded in monkey J when it was paired with monkey C (Fig. 3B). This M1 unit responded to both translational 
and rotational velocity and its firing increased during the ICS episodes. As before, this M1 unit also changed its 
velocity modulation properties when the monkeys’ roles were reassigned. When monkey J was the passenger, the 
unit increased firing rate in response to wheelchair rightward rotation combined with forward movement. When 
monkey J was the observer, the unit decreased firing rate to this pattern of wheelchair movement. 

For the entire neuronal sample, more units were modulated to the wheelchair velocity in the presence of syn- 
chrony episodes (39.0% + 5.0%, data from all monkeys combined) than in their absence (23.9% + 3.2%) (Fig. 3C, 
left). This difference was statistically significant for all monkey combinations (x7(1) =70.1, p < 0.05), and was not 
related to changes in absolute firing rate between the presence and absence of ICS (see Supplementary Materials: 
Controlling firing rate in ICS). The analysis of population-averaged modulation depth also showed a clear main 
effect of the presence of synchronization (ANOVA, F(1,1434) = 7.6, p <0.05) (Table S1; Fig. 3D, left). Thus, we 
observed both a significantly larger number of modulated units and an increase in modulation depth in the indi- 
vidual units during the ICS episodes. Additionally, ANOVA showed the main effect of monkey dominance rank. 
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Figure 3. Modulation of passenger’s and observer's units to wheelchair kinematics. (A) Modulation patterns 

to rotational (horizontal axis) and translational (vertical axis) velocity in a representative PMd unit recorded 

in monkey C. Color represents normalized (z-scored) firing rate. In these experiments, monkey C was paired 
with monkey K, and acted as passenger (left panels) or observer (right panels). ICS episodes were detected, and 
neuronal modulation was assessed separately when these episodes were present (ICS-+, top panels) and absent 
(ICS—, bottom panels). (B) Modulation patterns in an M1 unit recorded in monkey J. Monkey J was paired 
with monkey C. Conventions as in (B). (C) Bar plots representing average proportion of units modulated to the 
wheelchair velocity (left panel) and acceleration (right panel) for different monkey pairs, monkeys, and monkey 
roles (passenger or observer). Values are shown separately for the presence and absence of ICS episodes. Error 
bars represent 95% confidence interval obtained by 1,000 bootstrap replicates. (D) Averaged neural modulation 
depth to the wheelchair velocity (left panel) and acceleration (right panel). Conventions as in (C). 


In this analysis, monkey C had a higher dominance rank than the others. We found weaker neuronal modulations 
in the more dominant monkeys (F(1,1434) = 13.6, p< 0.05), and the modulation depth also depended on the 
monkey pairs (C-K or C-J pair, F(1,1434) =9.3, p <0.05). The analysis of neuronal modulations to acceleration 
showed that more units were modulated to acceleration in the presence of ICS episodes (19.3% + 1.2%; data from 
all monkeys combined) than in their absence (10.3% + 1.3%) (x7(1) =38.8, p < 0.05) (Fig. 3C, right). However, 
population-average modulation depth to wheelchair acceleration did not increase during the synchronous 
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Figure 4. Modulation of passenger’s and observer's units to the wheelchair room coordinates. (A) Modulation 
ofa PMd unit recorded in monkey C. Color represents normalized (z-scored) firing rate. In these experiments, 
monkey C was paired with monkey K, and acted as passenger (left panels) or observer (right panels). The 
observer was seated in the left corner (left panels) or right corner (right panels) (relative to the passenger at 
the starting location). Neuronal modulation was assessed separately when ICS episodes were present (ICS-+, 
top panels) or absent (ICS—, bottom panels). (B) Modulation of a PMd unit recorded in monkey K. Monkey 

K was paired with monkey C. Conventions as in (A). (C) Bar plots representing average proportion of units 
modulated to wheelchair position for different monkey pairs, monkeys, and monkey roles (passenger or 
observer). Proportions are shown separately for the presence and absence of ICS episodes. Error bars represent 
95% confidence interval obtained by 1,000 bootstrap replicates. (D) Averaged modulation depth. Conventions 
as in (C). 


episodes (ANOVA, F(1,1434) =0.03, p=0.86), and did not depend on the dominance rank (F(1,1434) =0.06, 
p=0.81) or monkey pair (F(1,1434) =0.39, p=0.53) (Table S2; Fig. 3D, right). 


Spatial modulation. M1 and PMd units were modulated to wheelchair room position in both the passen- 
ger’s and observer's brains (Fig. 4). The color plots of Fig. 4A,B show two representative PMd units whose firing 
rate changed when the passenger traveled to different room locations. These spatial modulation patterns were 
affected by the presence or absence of ICS episodes and monkey roles. Figure 4A shows a PMd unit recorded in 
monkey C when it was paired with monkey K. The neuronal rate increased with decreasing distance between the 
monkeys when monkey C was the passenger or observer. In both cases, the unit’s firing response to the distance 
between the monkeys increased during the ICS episodes (Fig. 4A, top vs bottom). The second illustrated unit 
was recorded in monkey K’s PMd. This unit’s rate decreased when monkey K was the passenger and approached 
monkey C (Fig. 4B, left). Conversely, the firing rate increased when monkey K approached the grape dispenser. 
‘The response pattern of the same unit was very different when monkey K was the observer (Fig. 4B, right). In this 
case, the neuronal rate increased when monkey K approached monkey C, but decreased when it approached the 
grape dispenser. For both the passenger and observer roles of monkey K, spatial modulation was stronger during 
the synchrony episodes (Fig. 4B, top vs bottom). 

Several analyses were conducted to assess different features of spatially-dependent modulations of neuronal 
rates. We first assessed neuronal rate as a function of room x,y coordinates. Next, we analyzed the dependence of 
firing rate on the distance between the monkeys. Lastly, we analyzed the role of the distance from the passenger 
to grape dispenser. In each analysis, we used the same ANOVA as in the previous section (see Methods: ANOVA 
for modulation depth). 

The analysis of modulation to x,y room coordinates showed that more units (x°(1) = 153.2, p < 0.05) were 
modulated to wheelchair position in the passenger (55.5% + 6.9%, data from all monkeys combined) com- 
pared to the observer (30.9% + 14.1%) (Fig. 4C; Table S9). Consistent with this result, average spatial modula- 
tion depth was higher in the passenger than observer in all cases (ANOVA, F(1,1434) = 88.1, p < 0.05; Methods: 
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ANOVA for modulation depth; Table S3). The presence of ICS episodes increased spatial modulation strength 
(F(1,1434) = 31.8, p< 0.05). When expressed as the percentage of spatially modulated units, this effect was sig- 
nificant for all monkey pairs (chi x?(1) = 179.0, p < 0.05): 66.7% + 2.5% of units (all monkeys combined) were 
significantly modulated to wheelchair location in the presence of ICS episodes and 43.8% + 5.2% in their absence. 
Additionally, modulation depth was lower for more dominant monkeys (F(1,1434) =8.6, p < 0.05) (Fig. 4D; 
Table S3). 

In terms of the responses to the distance between the monkeys, 47.6% + 4.3% of units (data from all mon- 
keys combined) were modulated in the passenger and 29.5% + 4.1% in the observer. The difference between 
these proportions was statistically significant (x?(1) = 102.0, p< 0.05). Modulation strength in the passenger was 
stronger than in the observer (ANOVA, F(1,1434) = 7.2, p < 0.05; Methods: ANOVA for modulation strength) 
(Table S4). The presence of ICS resulted in stronger modulation (F(1,1434) = 11.6, p < 0.05) (Fig. 5A). In addi- 
tion, another two ANOVA factors, social rank and role, also showed significant main effects: in a monkey pair, 
stronger modulations occurred in the monkey with a higher dominance rank (F(1,1434) = 12.1, p <0.05), and in 
the monkey that acted as observer (F(1,1434) =7.2, p < 0.05) (Table S4). Since it has been shown that mirror neu- 
rons are modulated differently to actions occurring in the extrapersonal and peripersonal space*, we conducted 
a separate examination of neuronal activity recorded when the inter-monkey distance was less than 1 m. For this 
distance range, neuronal rates significantly increased during the ICS episodes in both the passenger and observer 
(ANOVA, F = (1,1434) = 358.7, p < 0.05; see Methods: ANOVA for firing rates) (Fig. 5C; Table S5). The rates were 
lower for the dominant monkey in the pair (F = (1,1434) =7.2, p< 0.05), and units in the observer had higher 
rates compared to the passenger (F = (1,1434) = 43.6, p < 0.05). 

Modulation to distance between the passenger and the grape dispenser was found in 52.0% + 5.4% of the pas- 
senger’s units and in a small proportion (5.3% + 3.4%, x7(1) =418.0, p <0.05) of the observer’s units. Individual 
units were more strongly modulated in the passenger than the observer (ANOVA, F(1,1434) = 270.6, p < 0.05; 
Methods: ANOVA for modulation strength) (Fig. 5B; Table S6). Additionally, modulation strength increased dur- 
ing ICS episodes (F(1,1434) = 5.8, p < 0.05) (Fig. 5B; Table S6). For the 1-meter zone around the grape dispenser, 
neuronal rates were higher during the ICS episodes (ANOVA, F(1,1434) = 610.9, p < 0.05; see Methods: ANOVA 
for firing rates) and when the monkey was the subordinate monkey (F(1,1434) =54.3, p < 0.05) or the observer 
monkey (F(1,1434) =4.5, p < 0.05) (Fig. 5D; Table S7). 

Finally, we analyzed the changes in neuronal spatial modulation after the locations of the observer and grape 
dispenser were swapped. We tested whether the swapping symmetrically reflected the spatial modulation pat- 
tern relative to the longitudinal axis of the room. Our analysis showed that a symmetric reflection of the spatial 
modulation patterns occurred only for monkey pair C-K (average flip-index = 0.176 + 0.018, bootstrapping test, 
p <0.05; see Methods: flip-index), but not C-J (flip-index = —0.039 + 0.015, bootstrapping test, p< 0.05). The 
reflection was clearer for units recorded in monkey C, the most dominant monkey, (flip-index of 0.22 + 0.03) 
than monkey K (flip-index of 0.11 + 0.03) (bootstrapping test, p < 0.05) (Fig. 5E). The ANOVA for the flip index 
did not show differences in units’ flip-index for cortical area, monkey role, nor the presence of ICS (ANOVA, 
F(1,218) < 1.7, p > 0.19; see Methods: ANOVA for flip-index; Table S8). In monkey pair C-J, the pattern change 
could not be described as a symmetric reflection (flip-index = —0.039 + 0.015, bootstrapping test, p < 0.05), sug- 
gesting that cortical neuronal modulation was affected by the arrangement of the observer and dispenser posi- 
tions in room-centered coordinates. 


Discussion 

In this study, a whole-body navigation/observation task, combined with simultaneous wireless recordings of 
cortical neuronal ensemble activity from pairs of monkeys, was employed to investigate the neuronal correlates 
of spatial social interactions in primates. One monkey (the observer) remained stationary while another mon- 
key (the passenger) navigated using a robotic wheelchair. This paradigm allowed us to examine how the social 
interaction between the monkey pair was affected by whole-body movements of either the dominant or the sub- 
ordinate animals. Simultaneous recordings of cortical ensemble activity from both the passenger and observer 
revealed that their M1 and PMd units modulated their firing rate in accordance to the roles of the monkey (pas- 
senger or observer), as well as the wheelchair position and velocity. Moreover, we found that cortical units located 
in the two monkey brains exhibited episodes of transient synchronized firing. The probability and magnitude of 
such ICS depended on the wheelchair kinematics, the distance between the monkeys, and the distance between 
the passenger and the food reward. Based on these findings, we propose that high ICS defines a fundamental neu- 
rophysiological manifestation underlying social interactions in primates, and likely, other animals. 

Using concurrent wireless interbrain recordings, we identified the occurrence of episodes of social interaction 
even without considering behavioral measurements of such an interaction. In our approach, ICS was analyzed as 
a stochastic process, where the occurrence of synchronous episodes was described by conditional probability that 
depended on multiple parameters, including wheelchair position and velocity, inter-subject distance, and the dis- 
tance between the passenger and food reward. The color maps of the conditional probability of such ICS episodes 
were similar to classical neuronal tuning curves, with the key difference that our metric represented the combined 
activity of multiple brains rather than the activity of an individual brain. Such ICS maps allowed us to assess the 
role of experimentally controlled parameters in social interaction. Evidently, it is possible - and likely - that other 
uncontrolled factors contributed to the ICS observed here. The contribution from factors like eye/head move- 
ments, lip smacking, facial expressions, eye contact, other whole-body signals*~*’, and auditory responses (e.g., 
caused by subtle noise from the wheelchair motor) will have to be investigated in greater detail in future studies. 
These uncontrolled factors would only have a significant effect on neural activities if both monkeys attended to 
them simultaneously, and joint attention has been proposed as a basic mechanism for social interaction™. 
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Figure 5. Neuronal activity patterns across monkeys and ICS conditions. (A) Bar plots represent average 
modulation depth to the distance between the passenger and observer. Results are presented separately for 
different monkey pairs, monkeys, monkey roles (passenger or observer), and the presence/absence of ICS 
episodes. Error bars represent 95% confidence interval obtained by 1,000 bootstrap replicates. Panels (B-E) 
use the same conventions as in (A). (B) Average modulation depth to the distance between the passenger and 
grape dispenser. (C) Average normalized firing rates for the close (<1 m) distance between the passenger and 
observer. (D) Average normalized firing rates for the close (<1 m) distance between the passenger and grape 
dispenser. (E) Average flip-index for M1 (left) and PMd units (right). 


Mathematically, ICS can be described as an occurrence of specific neuronal firing patterns, like zero-lag syn- 
chrony*’, in a high-dimensional neuronal space composed of the activity of multiple brains. Here, we only con- 
sidered neuronal firing patterns generated jointly by two monkey brains, but in the future, the same analytical 
approach employed in the present study could be expanded to characterize meaningful and effective social inter- 
actions, from a systems neurophysiological point of view, that occur in large groups of primates, and likely, in 
other animal species, like rats and mice. Concurrent EEG and brain imaging can also be employed to study the 
same type of ICS in human subjects. In fact, several recent publications employed functional magnetic resonance 
imaging (fMRI) to assess brain processing in human subjects engaged in interactive and social behaviors”. The 
term “hyperscanning” was coined for such {MRI studies*’. In these studies, participants interacted simultane- 
ously or sequentially through interfaces, such as videos****°! and/or audios’. For example, one {MRI study” 
found a high level of synchronization between the subjects that viewed the same segment of a popular movie. 
Voxel-to-voxel synchronization was observed in the visual and auditory cortical areas, including primary, sec- 
ondary and association areas. Another study investigated neural correlates of verbal communications*’. {MRI 
recordings were first conducted in a speaker and then the audio was replayed while {MRI was obtained from a 
listener. Correlation analysis of the obtained brain scans suggested that successful verbal communication requires 
brain-to-brain coupling. Such coupling occurred when the speaker and listener shared the same language, and it 
diminished when the listener was told a story in a foreign language. Similar brain-to-brain coupling was demon- 
strated for non-verbal communications in humans, including communication with gestures*’ and facial expres- 
sions“. Interestingly, examination of the activity of each communicating brain alone failed to elucidate the extent 
of the neural circuitry involved in these interbrain interactions”. Recently, a dyadic imaging approach was devel- 
oped®™*4, where two participants laying side-by-side were scanned simultaneously in the same MRI scanner. 
Such a setting allowed face-to-face social interaction to be studied. In one study, brain activity obtained while 
two subjects placed in the same scanner established eye contact was compared with fMRI data obtained while a 
subject visualized pictures of opened/closed eyes of another person. This comparison revealed that establishing 
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eye contact with another person generates a very different pattern of brain activation than the one obtained when 
subjects just looked at pictures of human eyes. At this point, it is important to highlight that our study, as far as we 
can tell, is the first to employ concurrent wireless neuronal ensemble recordings in monkey dyads to demonstrate 
that ICS of cortical motor areas can be involved in representing multiple aspects of primate social interactions. 

Additional knowledge on brain-to- brain coupling has been provided by brain-machine interface (BMI) stud- 
ies where neural activity was simultaneously decoded from the brains of a group of subjects trained to control 
various external devices jointly. The term “Brainet” was recently coined by our laboratory to describe BMI imple- 
mentations of this kind>. Historically, such multi-brain systems were first implemented in the 1970s using elec- 
troencephalographic (EEG) recordings in humans*!”*. These early applications enabled a variety of cooperative 
tasks, including EEG-controlled music performance by multiple subjects*®°’ and drawing of Lissajous curves by 
two participants synchronizing their alpha EEG rhythms”. In our lab, intracranial recordings from ensembles of 
monkey cortical neurons were utilized for a similar purpose: pairs or trios of monkeys learned to cooperatively 
control the reaching movements performed by a virtual arm using their combined cortical ensemble activity”. 

Several factors contributed to the occurrence of ICS observed in our study. First, wheelchair movements 
triggered episodes of ICS because they simultaneously engaged cortical processing in both monkeys. When 
the wheelchair moved, the passenger’s M1 and PMd units responded to the animal’s whole-body displace- 
ment. Conversely, in the observer, equivalent cortical neuronal populations responded as a way to represent the 
passenger’s movements in a mirror fashion. In addition, the distance between the monkeys was an important 
social-interaction factor that affected the probability of the two monkey motor cortices being synchronized. 

Furthermore, cortical activity in both monkeys’ brains was related to the expectation of simultaneously 
acquiring rewards, when the passenger monkey reached the region where it could collect grapes and the observer 
received a fruit juice reward. As such, this reward expectation defined another important way in which the mon- 
key pairs interacted socially. Consequently, the distance between the passenger and reward was found to be corre- 
lated with the probability of ICS episodes. Although previous studies have considered behavioral manifestations 
of similar social-interaction factors, including joint displacement and relative position of group members” 
and food seeking and consumption””*, our study is the first to document the occurrence of ICS in at least two 
cortical motor areas, M1 and PMD, as the potential neurophysiological manifestations underlying such social 
interactions. In this context, our present finding extends previous results obtained in individual monkeys that 
showed that M1 and PMd neuronal ensembles also encode reward magnitude and expectation®*®°. Here we 
showed that potential rewards are also encoded by ICS. Since responses to reward in the acting monkey likely 
represented dopaminergic inputs to M1 and PMd™, it is reasonable to suggest that similar dopaminergic effects 
could be present in the observer's brain in our task. Taking this argument further, one could raise the hypothesis 
that dopaminergic effects during the observation of rewarded actions may explain the phenomenon of learning 
by observation®®*”. In the context of the present study, ICS could be interpreted as a potential neuronal mani- 
festation of social learning and even a mechanism to facilitate knowledge transfer from one subject to the other. 

Interestingly, ICS also reflected high-order variables involved in social interactions, such as the composition 
of monkey pairs and their assigned roles in the task. Indeed, the magnitude of the ICS in a monkey pair changed 
when a passenger and an observer flipped their roles. For example, strong ICS was observed when monkey C 
and monkey K were close to each other and C was the passenger. Conversely, this ICS decreased when monkey 
C became the observer. These results could be related to the social hierarchy of the monkeys in our colony. In 
support of this suggestion, it was observed in previous studies that dominant monkeys freely roam in their sur- 
rounding space, while submissive monkeys suppress their behaviors in that space to avoid conflict”°™**. In our 
experiment, monkey C’s dominance rank was higher than monkey K and J. This difference in the monkey’s social 
rank could translate into the effects we observed here. Irrespective of the social ranking of our monkey colony, it is 
reasonable to suggest that the one-meter space around the observer could be considered as “the zone of potential 
conflict” for the monkey pairs. This could explain the occurrence of prominent ICS when the passenger entered 
this zone. 

Although ICS obviously was not related to direct connectivity between the monkeys’ brains, it is possible that 
the Hebbian rule “fire together, wire together” is a valid metaphor for describing the plastic neurophysiological 
changes resulting from continuous social interaction. Using this analogy, the enhancement of “social connectiv- 
ity” between multiple animal brains, instead of synaptic connectivity, may be strengthened or weakened, through 
changes in ICS resulting from social contact. For example, if the passenger’s whole-body movement, associated 
with bursts of activity in the passenger's M1 and PMd neurons, is consistently coupled with modulations in firing 
rate of the observer's equivalent neurons, the two monkey brains get “effectively synchronized’, although they are 
not interconnected directly. In real-life situations, such social connectivity is likely to result in causal relation- 
ships. For example, cortical activity of a monkey engaged in a behavior would translate into the cortical activity of 
a nearby monkey, and so on, until the repetition of this social interaction could effectively reshape the interbrain 
functional connectivity of an entire social group. Our lab has called the resulting chain of synchronized brains a 
Brainet*’. According to our view, the occurrence of strong episodes of ICS would be the major determinant lead- 
ing to the creation of such Brainets through animal social interactions occurring in their natural environments. 
In this context, such Brainets could include a large number of individual brains. 

Given this new view on how social interactions are represented by the combined activity of multiple brains 
and how ICS could underlie social learning, it would be of interest to compare our findings with the previous 
theoretical framework built around the notion of mirror neurons. Several observations in our study are rem- 
iniscent of previous findings related to mirror neuron activity. For example, the observation that a significant 
fraction of M1 and PMd units in the observer's brain are modulated to the wheelchair’s rotation and translational 
velocity is consistent with the literature on mirror neurons representing observation of actions in premotor**!*!° 
and motor®” cortical areas. Mirror neurons are described as cortical neurons that respond the same way when 
a subject performs or observes an action. This definition is only marginally applicable to our findings because, 


SCIENTIFIC REPORTS | (2018) 8:4699 | DOI:10.1038/s41598-018-22679-x 9 


www.nature.com/scientificreports/ 


generally, a given PMd or M1 neuron had different tuning patterns during the monkey’s navigating or observing. 
Yet, the modulation depth for navigation was correlated with the modulation depth for observation, suggesting 
the existence of typical mirror neuron activity. Additionally, average modulation depth was higher in the pas- 
senger than the observer. While it is tempting to attribute this result to a stronger representation of self-motion 
in M1 and PM, as opposed to motion observation, it is also possible that the observer was not attentive enough 
to the passenger’s movements in our experiments. Indeed, paying attention to the passenger was not required 
by the task. It would be of interest in the future to employ a behavioral task where the observer must attend to 
the passenger to obtain the reward. In one previous study”, monkeys were required to attend to the movements 
performed by a robot, and a significant portion of PMd neurons (~20%) became modulated to spatial attention 
orientation. 

The other result that somewhat resembles previous findings on mirror neurons is our demonstration of neu- 
ronal tuning to the distance between the passenger and observer. A previous study has already demonstrated 
that the distance between an observer (a monkey) and an actor (a human performing a motor task in front of 
the monkey) influences mirror activity in premotor cortex*. In our experiments, M1 and PMd neurons in both 
the passenger and observer were modulated to the distance between the monkeys, especially when the passen- 
ger entered the 1-m zone surrounding the observer. Interestingly, swapping the passenger and observer often 
changed the pattern of this distance tuning, which indicated that the neuronal representation of “someone enter- 
ing my space” was different from the representation of “me entering someone’s space”. 

Our finding of distance-tuning to reward also bears resemblance to previous reports of mirror-like activity 
related to observation of ingestive actions®!-®>”!-”°, Here, we demonstrated that both the passenger’s and observ- 
ers M1 and PMd units were modulated to the distance from the passenger to reward. As in the case of tuning 
to inter-monkey distance, reward-related tuning patterns depended on the monkey pair composition and the 
assigned monkey roles. This finding suggests that monkey dominance ranks, defined by preferential access to 
food and aggression”, played a role in the cortical encoding of reward location. 

Despite being generally consistent with the mirror-neuron framework, our findings clearly go beyond this 
classical paradigm by describing, for the first time, a potential neuronal mechanism underlying social interaction 
in the form of episodic ICS involving multiple motor cortical areas. Previously, classic sensorimotor physiology 
described cortical motor control and associated sensory processing as activities confined to distinct cortical areas 
located in a single brain. For example, in an individual animal, cortical motor areas would be involved with the 
planning of an arm movement and then contribute to its execution, while adjusting the movement based on 
sensory feedback. The finding of mirror neurons adds to this view the notion that the same cortical areas that 
control movements also respond to the observation of movements performed by others. These descriptions apply 
to two types of behavior: (1) production of actions, and (2) observation of actions; but do not integrate them. 
Social interaction is a type of behavior where actions are amalgamated with observations. Accordingly, our dis- 
covery that social interaction between pairs of monkeys can be represented by widespread ICS merges both action 
and observation as part of a common neurophysiological interactive process, taking place simultaneously in the 
motor cortical areas of multiple primate brains interacting as part of a social group. Therefore, we further propose 
that studying the patterns of such ICS will increase our knowledge of movements that are planned and executed in 
the context of the rich social interactions that characterize the lives of primate and other animal groups, including 
humans. This new view indicates the need to reconsider the role of motor cortical areas to include their involve- 
ment in animal social interactions and how the latter influence the moment to moment operation of such cortical 
motor circuits. 

Overall, the present demonstration of neural correlates of social interaction in the form of ICS has implica- 
tions for future clinical application as well, especially for disorders that include social interaction deficits, such as 
autism spectrum disorders, which may involve difficulties in representing/understanding the actions of others, 
while generating appropriate social behaviors’*”°. Previously, these conditions have been linked to disorders of 
the brain mirror system’*”’. Our current findings are relevant to this framework and add to it by showing that 
social interactions may be encoded by the episodes of ICS. Accordingly, we suggest that our approach could be 
used as a diagnostic tool for detecting abnormal interbrain activities during human social interactions. In addi- 
tion to being a potential biomarker for quantifying the severity of different forms of autism, measurement of ICS 
could be used as a tool for monitoring the effects of autism treatment, like behavior therapy, and also as part of a 
neurofeedback system for improving social motor skills, like in high-performance collective sports. 


Methods 
All animal procedures were performed in accordance with the National Research Council’s Guide for the Care 
and Use of Laboratory Animals and were approved by the Duke University Institutional Animal Care and Use 
Committee. 


Study Design. Three adult rhesus macaques (monkey C, K, and J) were used in this study. They were chroni- 
cally implanted with microwire arrays in multiple cortical areas (monkeys C and K in January 2012, and monkey 
J in January 2017). In monkey K, we recorded from M1 and PMd; in monkey C, from M1 and PMd; and in mon- 
key J, from M1. Neuronal spiking activity was recorded using a wireless recording system developed in-house 
that samples from 128 channels simultaneously in each monkey. A pair of monkeys (C and K, or C and J) was 
placed in the experimental room. One monkey (the observer) sat in a stationary chair while the other monkey 
(passenger) sat in the robotic wheelchair that navigated in the room. The room size was 5.0-by-3.9 m. The wheel- 
chair navigated in the 3.5-by-2.6 m part of that room. The maximum translational and rotational velocities of 
the wheelchair were 0.3 m/s and 0.5 rad/s, respectively, to ensure the comfort of the passenger. When placed in 
the initial position, the passenger faced the observer in one corner of the room and a food dispenser in another 
corner. The positions of the observer and the food dispenser were swapped in different experiments. After the 
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passenger arrived at the dispenser’s location, the dispenser dropped a piece of fruit (grape, apple, blueberry, or 
carrot) on a plate mounted on a robotic arm, the arm extended towards the observer, and the observer grasped 
the food. 


Behavioral task. The robotic wheelchair carried the passenger from a randomly generated starting position 
toward the food dispenser. The navigation trajectory passed through two randomly generated checkpoints before 
arriving at the food dispenser location. After the passenger retrieved the food, the wheelchair autonomously 
navigated back to a new randomly generated starting position. During the wheelchair navigation toward the 
dispenser, the observer monkey received juice rewards: small drops of juice were delivered every 10-305, but 
only if the observer's head pointed in the direction the room center. A large juice reward was delivered when the 
wheelchair arrived at the food dispenser. 


Social ranking. We assessed the social hierarchy in monkey pair C-K and C-J using a paradigm where they 
competed for food’”*!. The two monkeys sat in chairs and faced each other at the distance where they could reach 
for a small piece of fruit (grape, apple, or strawberry) placed on a tray. We conducted 50 trials, where the food was 
placed in between the monkeys, and we counted how often each of the monkeys reached out first. For the pair 
C-K, monkey C and K obtained the fruit 32 and 18 times each 2(1) =7.8, p<0.05); for monkey pair C-J, mon- 
key Cand J obtained the fruit 36 and 14 times each (x? (1) =19.4, p < 0.05). These results suggested that monkey 
C was the dominant monkey in each pair. 


System integration. The experiment setup included three components: (1) the experiment control sys- 
tem, (2) the wheelchair navigation system, and (3) the wireless recording system. The experiment control system 
supervised the sequence of task events. The autonomous wheelchair navigation system controlled the wheelchair 
and reported the position of the wheelchair to the experiment control system. The wireless recording system 
recorded neuronal ensemble activity from two monkey brains simultaneously and sent the spike timestamps to 
the experiment control system. The three systems communicated using a local network. 


Experiment Control System. The experiment control system controlled the task sequence, including starting 
a trial, setting target locations for the wheelchair, determining whether the wheelchair has reached the target, 
delivering food and liquid rewards, and ending a trial. This system received the wheelchair coordinates from the 
wheelchair navigation system at 10 Hz and sent target locations to the wheelchair navigation system. The experi- 
mental control system also received multichannel neuronal data from the wireless recording system. 


Wheelchair Navigation System. ‘To move the wheelchair from one location to another, the robust autonomous 
navigation was implemented. The wheelchair was equipped with a Roboteq VDC2450 dual channel motor con- 
troller and wheel encoders to provide closed-loop control and odometry. A lidar (RPLidar 360 Laser Scanner by 
Robopeak) was installed at the front side of the wheelchair to sense the distance to its surroundings. The motor 
controller and the lidar were interfaced via a local wired/wireless network with a Raspberry Pi (RP), which com- 
municated with the computer that ran the experiment control system. 

We used Robotic Operating System (ROS) software to provide the wheelchair functionality, including autono- 
mous navigation, obstacle avoidance, simultaneous localization and mapping (SLAM). ROS ran on two comput- 
ers: the RP and a dedicated desktop computer for navigation. The two computers communicated through ROS 
topics, on which one computer could publish messages and the other could subscribe. To localize the wheelchair, 
a map of the experiment room (Fig. $3) was first generated by Hector SLAM® before the very first session, and 
this map was used for all the sessions. Then, combining sensor data published by the RP and wheelchair velocity 
commands published by the navigation computer, a particle filter approach was used to localize the wheelchair 
at 10 Hz. Given the position of the wheelchair and the navigation destinations from the experiment control com- 
puter, the navigation computer computed and published the wheelchair velocity commands within the predefined 
ranges (0 to 0.3 m/sec for translations, and —0.5 to 0.5 rad/s for rotations) at 20 Hz (ROS navigation package), 
which the RP subscribed and passed to the motor controller for execution. 


Wireless Recording System. The wireless recording system was built in-house, as described in**”». In short, it was 
composed of two wireless headstages of 128-channels (one for each monkey), two bridge receivers (one for each 
headstage), and one recording computer. Once spikes were sorted in the recording computer, the 16-point spike 
waveform templates were transmitted to the storage on the headstage. The headstage sampled neural activities 
of each channel at 31,250 Hz, and the headstage performed spike sorting on an FPGA using a template algo- 
rithm, where a spike was detected if the absolute distance between the waveform and the template was within a 
user-specified threshold. Detected spike occurrences and channel IDs were wirelessly sent to the bridge receiver 
and then routed to the recording computer through wired ethernet. The recording computer timed the detected 
spikes (temporal resolution was estimated about 5 ms) and stored this information and sent neuronal data to the 
system control computer for further processing, such as online decoding of neural signals. 


Data analysis. We conducted seven sessions with the monkey pair C-K and four sessions with the pair C-J. 
For C-K, each session lasted 99.3 + 17.3 (mean + standard deviation) 99.3 minutes and consisted of 80 +45 tri- 
als. Only the movements of the wheelchair toward the food dispenser (62.8 + 10.4 minutes) were analyzed; the 
returns to the starting positions were excluded from the analysis. Each trial began when the wheelchair started 
at the starting positions and ended when the wheelchair arrived the target location, where the fruit has not being 
presented to the monkey at the time so that no reaching related activities was included in the analysis. For the 
first three experimental sessions, monkey K was the passenger, and monkey C was the passenger during the next 
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three sessions. On the last, the seventh session, monkey K was the passenger again. The observer remained in the 
same corner of the room for the first five sessions, and then the locations of the observer and grape dispenser were 
swapped. For the C-J pair, the four sessions lasted on average 55.9 minutes (std. 6.4 minutes) and included 48 tri- 
als (std. 1.3 trials). The movements of the wheelchair toward the dispenser lasted 31.5 minutes (std. 0.6 minutes). 
Monkey C was the passenger for the first two sessions, then monkey J became the passenger for the next two 
sessions. The observer was located in one room corner for the first and fourth sessions, and the opposite corner 
for the second and third sessions. 


Modulation depth. Modulation depth was calculated based on linear regression. We sampled the variates (e.g., 
wheelchair position) with a 100-ms sampling interval within the interval 500 ms before and after time t and we 
regressed them to the value of neural firing rate at time t. Once the regression model was trained, we computed 
correlation coefficient between the true and the fitted firing rate time series as the uncorrected modulation depth 
(r’). 

To test whether a unit was significantly modulated to a variate, we compared r’ against the modulation depth 
r” computed from randomly sampled and permuted firing rate series. We performed the permutation test with 
1,000 permutations, and the p-values were corrected by false-discovery rate. We considered a unit significantly 
modulated if the corrected p-value was under 0.05. Note that for each ICS condition, only the firing rate time 
series that belonged to that condition was permuted and tested against. 

To compare neuronal modulation under different ICS conditions, the modulation depth r was computed from 
the same length of data across conditions, where the length was the duration of synchronized episodes for the 
whole session. Finally, we computed unbiased modulation depth, r = r’ — mean(r”), r was used as the metric for 
modulation depth in all analyses. 


ANOVA for modulation depth. We assessed the effect of the presence of synchrony episodes (ICS+ vs. ICS—), 
monkey role (passenger vs. observer), social rank (dominant vs. subordinate), and monkey pair (C-K vs. C-J) 
on modulation depth using a mixed-designed four-way ANOVA, where ICS episode was a repeated measure 
obtained from the same unit, while others were non-repeated measures. Modulation depth of each unit was 
treated as the random effect, while all others were fixed effect. Sums of squares of type 3 were employed when the 
data was unbalanced. 


ANOVA for firing rates. ANOVA was designed the same way as that for modulation depth. 


ICS metric. Bias-corrected distance correlation was originally proposed by Székely and Rizzo*! to quantify the 
strength of correlation between two random vectors in arbitrarily high dimensions that do not need to be equal. 
We chose distance correlation over RV coefficient because distance correlation is more sensitive to detect nonlin- 
ear dependency®. The distance correlation was computed as R*(X, Y) = V2", where V%(X, Y) is the 


[Vi XVe(y, Y) 


modified distance covariance between two random vectors X € R? and Y € R’. This distance correlation test 
statistic has an asymptotic student t distribution under independence, and thus t-test is used for multivariate 
independence in high dimension. Also, R* is always positive if X and Y are correlated. 

Brain-to-brain correlation was computed as bias-corrected distance correlation between two random vec- 
tors, where each random vector was the neural firing rate (0.1 s time bins) of the population of units from each 
monkey. When analyzing episodic brain-to-brain correlation, the correlation was computed within a 3-s sliding 
window shifted with 0.1 steps. 


Episodic ICS. To test whether an ICS episode was significant, we first computed interbrain correlation for all 
0.1s time steps (see ICS metric above). Then, we permuted the spike trains from one monkey and computed the 
correlation again for the permuted data, which resulted in a null distribution of ICS (Fig. S1). Lastly, we tested ICS 
value of each 3s sliding window against this null distribution to determine whether it is a significant ICS episode 
(t-test with alpha = 0.05 for a right-tail test, and p-value was corrected by false discovery rate). 


ICS Probability. ICS probability was calculated as the conditional probability of the occurrence of an ICS epi- 
sode given the wheelchair position or velocity. To compute the ICS probability as a function of room coordinates 
of the wheelchair, the room coordinates were binned into a 0.2 m? grid, where x spanned from 0.1 m to 3.1 m and 
y spanned from —0.8m to 1.2m. The ICS probability was computed for each bin as the count of ICS episodes, 
divided by the total count of the wheelchair entering the bin. Similarly, the passenger-grape distance and the 
passenger- observer distance were both binned from 0.4 m to 4m with 0.2m step. To compute the ICS probability 
as a function of the wheelchair kinematics, the rotational velocity was binned from —0.5 to 0.5 rad/s with bin size 
of 0.05 rad/s; the translational velocity was binned from 0 to 0.3 m/s with bin size of 0.05 m/s. 


Decoder output correlation. Correlation between the decoder outputs was computed as the bias-corrected dis- 
tance correlation between the decoded velocities decoded for each monkey. This correlation was also computed 
within 3-s sliding window shift with 0.1s steps. 


Flip-index. Flip-index measured if the neuronal modulation pattern to different room locations symmetrically 
reflected when the locations of the food dispenser and the observer were swapped. We first computed pixel-wise 
Spearman correlation between the spatial modulation patterns being compared. Then, we flipped one of the 
patterns and computed the correlation coefficient again. The flip index of a unit was the difference between these 
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two correlation coefficient values. Note that, because the food dispenser and the observer only swapped locations 
between different sessions, we assumed units of the same ID number from different sessions were the same units 
for this analysis (see Supplementary Fig. S5). Also, the flip index was calculated between sessions where the mon- 
keys had the same roles. 


ANOVA for flip-index. Multi-factor ANOVA assessed the effect of the presence of synchrony episodes (ICS+ vs. 
ICS—), monkey role (passenger vs. observer), recorded area (M1 vs. PMd), and social rank (dominant vs. subor- 
dinate) on flip-index. We used a mixed-designed four-way ANOVA. We treated synchrony episodes as repeated 
measures from the same units, while the others as non-repeated measures. Flip-index was the only random effect, 
while others were fixed effect. Sums of squares of type 3 were employed when the data was unbalanced. 
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Although Donald Hebb’s classic book, The Organization 
of Behavior (1949), is widely known for its description 
of a mechanism for synaptic plasticity (the so-called 
Hebbian synapse), it also contains one of the mostinflu- 
ential proposals on how interactions between large pop- 
ulations of neurons underlie brain processes. Central to 
Hebb’s theory is the concept of the ‘‘cell assembly... 
a diffuse structure comprising cells in the cortex and 
diencephalon, capable of acting briefly as a closed sys- 
tem, delivering facilitation to other such systems .. .” 
(p. xix). In Hebb’s view, individual neurons did not work 
in isolation, and consequently, they could not, by them- 
selves, account for any given percept or ability. In fact, 
he suggested that individual neurons could participate 
in different cell assemblies and be involved in multiple 
functions and representations. Although the importance 
of neural populations in sensorimotor information pro- 
cessing had been recognized earlier (Young, 1802; Sher- 
rington, 1906), Hebb’s work was a landmark because it 
provided the first elaborated description of mechanisms 
by which neural populations could underlie a variety of 
brain functions. 

Because single neurons can respond to a range of 
sensory stimuli or participate in multiple motor acts, 
large neural populations are required for representing 
the different attributes of a particular stimulus or for 
producing a given behavior. Population coding schemes 
have been proposed for several sensory modalities (re- 
viewed by Erickson, 1968), as well as for cortical control 
ofarm movements (Georgopolous et al., 1986) and tectal 
control of saccadic eye movements (Lee et al., 1988). 
Nevertheless, until recently, the main approach used 
to reconstruct both sensory and motor representations 
was to record, ina Serial fashion, the activity ofindividual 
neurons and then try to derive a population code. Unfor- 
tunately, this approach does not allow one to investigate 
the potential time-dependent interactions between neu- 
rons that may be used by the brain to represent infor- 
mation. 

The recent advent of new electrophysiological tech- 
niques, which currently allow one to record the simulta- 
neous activity of 100-150 neurons, has sparked renewed 
interest in the properties of neural assemblies and their 
potential roles in brain function. Therefore, it is not sur- 
prising that many laboratories have begun applying neu- 
ral ensemble recordings to investigate how neuronal 
populations encode sensory and motor information. 
Here, we examine some of this recent work, which sug- 
gests that Hebb’s view is likely to become the rule rather 
than the exception. 

Neural Ensemble Encoding of Sensory Information 
Temporal interactions between cell assembles in differ- 
ent anatomical locations were a fundamental postulate 


Minireview 


of Hebb’s theory. Testing this theory would require one 
to record from populations of neurons distributed across 
multiple cortical areas and subcortical nuclei. Using a 
chronic multiple-electrode recording preparation, Nico- 
lelis and coworkers (1995, 1997a) recorded from popula- 
tions of neurons distributed throughout the trigeminal 
somatosensory pathway, from the trigeminal brain stem 
complex and the somatosensory thalamus to the pri- 
mary somatosensory cortex, in awake, behaving rats. 
These authors showed thatensembles of single neurons 
from most of these structures exhibited widespread, 
synchronous oscillatory firing that began during atten- 
tive immobility and predicted the onset of rhythmic 
whisker movements. These oscillations were detected 
first in cortex and then spread to subcortical structures. 
The cortical and subcortical ensembles underlying such 
rhythmic firing have been found to contain highly distrib- 
uted representations of tactile information (Ghazanfar 
and Nicolelis, 1997; Nicolelis etal., 1997a, 1997b). Multi- 
variate statistical analysis (e.g., discriminant analysis 
and canonical correlation) revealed that in this sensory 
system, the precise location of a tactile stimulus could 
only be unambiguously predicted, ona single trial basis, 
when population rather than single neural responses 
were taken into account (Nicolelis et al., 1997b). There- 
fore, these results emphasized that the coordinated ac- 
tivity of large ensembles of neurons, distributed across 
cortical and subcortical structures, may provide the ba- 
sis for the encoding of tactile information in mammalian 
somatosensory systems. 

Maldonado and Gerstein (1996) were interested in de- 
termining the changes in neuronal ensemble dynamics 
that follow sensory reorganization induced by intracorti- 
cal microstimulation in the auditory cortex of the rat. 
Intracortical microstimulation is known to produce a 
broadening of the receptive fields of cells located at 
the stimulation site. In addition, neurons recorded from 
adjacent electrodes have been shown to increase their 
responsiveness to the best frequency of the cells re- 
corded from the stimulating electrode. The functional 
relationships between neurons distributed within the pri- 
mary auditory cortex were assessed using gravity analy- 
sis, a method in which the temporal relationships be- 
tween neuronal spike trains are represented as a Series 
of clusters in a multidimensional space. In this multidi- 
mensional space, neurons attract or repel each other 
depending on the coincidence of their neuronal firing. 
The responses of up to eight neurons were recorded 
simultaneously following auditory stimuli and intracorti- 
cal microstimulation. These experiments demonstrated 
that the functional clustering of a subset of the simulta- 
neously recorded neurons, obtained during the delivery 
of the auditory stimuli, could be strengthened following 
intracortical microstimulation. The formation of a func- 
tional cluster of neurons did not necessarily relate to the 
anatomical distance between the cells. In other words, 
neurons that were anatomically close did not necessarily 
have a strong interaction, and neurons that were far 
apart did not necessarily have weak interactions. These 
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results showed that neural ensembles can be estab- 
lished transiently, that they are not necessarily com- 
posed of neurons within a circumscribed location (e.g., 
a cortical column), and that membership in an ensemble 
is mutable as a function of the induced reorganization. 
Time-dependent encoding of sensory information has 
also been described in the locust olfactory system. In 
a series of elegant studies, Laurent and colleagues 
(Wehr and Laurent, 1996; Laurent et al., 1996) recorded 
from ensembles of two to five projection neurons in the 
antennal lobe of the locust during the presentation of 
various odors to the animal’s antenna. Several important 
findings emerged from these experiments: 1) multiple 
neurons, distributed throughout the lobe, responded 
during the presentation of the same odor; 2) different 
odors could elicit unique responses from a given neuron; 
and 3)neurons that responded during a given odor pre- 
sentation did so during specific epochs of the response, 
corresponding to cycles of field potential oscillations 
in the mushroom body, which receives input from the 
antennal lobe projection neurons. Thus, neural assem- 
blies in the antennal lobe respond to each odor with a 
unique spatiotemporal pattern of firing. 
Coding of Task Parameters by 
Hippocampal Ensembles 
The hippocampus has long been implicated as a major 
component of a system that is involved in memory and 
in the representation of spatial information. In a recent 
study, Deadwyler et al. (1996) investigated how activity 
in simultaneously recorded CA1 and CA3 ensembles (10 
neurons per ensemble) concurrently encodes several 
task-related events in a delayed-non-match-to-sample 
lever-press paradigm. Central to this investigation was 
the use by these authors of discriminant analysis and 
canonical correlation, which has been adapted for simul- 
taneously recorded neuronal population data sets (Nico- 
lelis et al., 1997b). It was shown that the spatiotemporal 
patterns of hippocampal ensemble firing encoded four 
different task-related parameters: the phase of the task 
(i.e., sample versus non-match phase), errors committed 
on the non-match phase of the task, the position of 
the lever being pressed, and the position of the lever 
presented in the sample phase ofthe task. Itis important 
to note that while the firing patterns in the hippocampal 
cells differed from animal to animal, the same four task 
parameters could be extracted from the ensemble firing 
patterns in all animals. This indicates that the same 
dimensions of the behavioral task were encoded by the 
neuronal ensembles even though their firing patterns 
were not the same, and emphasizes that the activity of 
individual neurons is not sufficient for encoding these 
types of behavioral parameters. Instead, as Hebb postu- 
lated, patterns of activity must be analyzed across en- 
sembles of neurons to determine how such information 
is represented. 
Dynamic Encoding of Motor Behavior 
In their pioneering work on the primate motor cortex, 
Georgopolous and coworkers (1986) elegantly demon- 
strated that populations of neurons could accurately 
predict the trajectory of arm movements. In their studies, 
a neural population vector was derived by pooling to- 
gether the responses of serially recorded single units in 
different recording sessions and in different animals. 


Although these vectors could be used to predict the 
direction of arm movement, the potential of using time 
as a coding dimension was lost by such an approach. 
This is a relevant issue, since it is conceivable that the 
same population of cortical motor neurons could use 
time-dependent coding schemes to represent different 
attributes of motor behavior. 

The importance of the time domain in cortical motor 
coding has recently been demonstrated by Abeles and 
colleagues who obtained simultaneous recordings of 
six to eight neurons in the primate frontal cortex while 
animals performed a delayed-localization task (Abeles 
et al., 1995; Seidemann et al., 1996). This task required 
the animals to make arm movements to a remembered 
visual target that was flashed either left or right of a 
reference light. By implementing a hidden Markov model 
to analyze the simultaneously recorded neuronal spike 
trains, these authors proposed that the cortical neural 
ensembles go through a sequence of discrete, stable 
states during the delay period of the task (when the 
monkey must remember the target location). These sta- 
ble states were characterized by a specific station- 
ary pattern of relative firing between neurons, which 
changed abruptly from one state to another. In addition, 
these states were not time locked to the occurrence of 
any specific sensory or motor event, and the particular 
sequence of states could be used to predict, with ~90% 
accuracy, the response of the monkey. If the correlated 
firing-rate modulations were eliminated from the ensem- 
ble activity, but the overall firing rate was preserved, no 
clear states or sharp transitions between states could 
be detected by the hidden Markov model. Similarly, if 
ensembles were formed by neurons recorded serially, 
the hidden Markov model failed to detect discrete and 
stable states of neuronal ensemble activity. This work 
underscores the importance of the temporal domain in 
ensemble coding and suggests that fundamental infor- 
mation processing at the level of cell assemblies may 
occur even in the absence of a particular sensory stimu- 
lus or motor output. 

Cell assembly encoding of motor output patterns has 
also been studied extensively in several invertebrate 
systems. This work has shown that neurons within a 
single ganglion can participate in multiple networks that 
yield different behaviors at different moments in time. 
Forexample, ithas been shown thatsome crab stomato- 
gastric ganglion motor neurons can participate in sepa- 
rate feeding rhythms, the gastric and pyloric rhythms 
(Wieman etal., 1991). Thus, these neurons do not belong 
to unique stomatogastric ganglion networks, but in- 
stead, can change their activity to participate in both 
motor output patterns. Similarly, Wu et al. (1994) used 
optical recordings to simultaneously sample the activity 
of multiple neurons in the abdominal ganglion of Aplysia. 
In this preparation, large groups of neurons were acti- 
vated during three gill-related motor behaviors: the gill 
withdrawal reflex, spontaneous gill contraction, and re- 
Spiratory pumping. These motor activities were not en- 
coded by dedicated circuits, butinstead were controlled 
by distributed networks of neurons, whose members 
were selected from a larger neuronal population that 
participates in the genesis of multiple behaviors. 
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Future Directions 

What we are witnessing in modern neurophysiology is 
increasing empirical support for Hebb’s views on the 
neural basis of behavior. While there is much more to 
be learned about the nature of distributed processing 
in the nervous system, it is safe to say that the observa- 
tions made in the last 5 years are likely to change the 
focus of systems neuroscience from the single neuron 
to neural ensembles. Fundamental to this shift will be 
the development of powerful analytical tools that allow 
the characterization of the encoding algorithms em- 
ployed by distinct neural populations. Currently, this is 
an area of research that is rapidly evolving. 

Further demonstration of a causal link between neural 
ensemble activity patterns and specific sensations or 
behaviors is necessary to demonstrate the relevance 
of population coding in the CNS. This issue is being 
approached in several ways. On one hand, information 
obtained at the molecular and cellular level is beginning 
to be applied to the investigation of circuit properties. 
For instance, ensemble recordings can now be com- 
bined with other neurobiological approaches, such as 
knockout genetics and/or the selective elimination of 
specific cell types (e.g., McHugh et al., 1996). These 
techniques will allow us to investigate what role a spe- 
cific cellular population may play in information coding 
by large cell assemblies. At the other end of the spec- 
trum, chronic and simultaneous multisite neural ensem- 
ble recordings can now be performed in behaving pri- 
mates (Nicolelis et al., 1996, Soc. Neurosci. abstract). 
Since these recordings remain stable for many months, 
this opens the possibility of investigating how the learn- 
ing of sensorimotor or cognitive tasks impacts the large- 
scale neuronal interactions within and between cortical 
and subcortical neural ensembles. These and other ex- 
citing developments promise to open a new era of inves- 
tigation in systems neuroscience. 
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Real-time direct interfaces between the brain and electronic and mechanical 
devices could one day be used to restore sensory and motor functions lost 
through injury or disease. Hybrid brain-machine interfaces also have the 
potential to enhance our perceptual, motor and cognitive capabilities by 
revolutionizing the way we use computers and interact with remote 


environments. 


After a clever throw in by Tostao, a simple flick of Rivelino’s magic left foot was enough to send 
the ball soaring into the thin air of the Azteca stadium in Mexico City. As the immaculate white 
object flew towards the middle of the penalty box on that hot afternoon, the colourful crowd that 
packed the stands slowly rose in anticipation. They roared, already celebrating, because they 
had seen that scene a thousand times before: the same graceful black man, dressed in blue shorts 
and a yellowjersey with the green 10 sewn in the back, defying logic, making fun of physics. The 
early celebration was warranted. As expected, Pelé floated above all Italian defenders to 
encounter the ballin mid air, and, witha gentle kiss of a forehead, changed its trajectory towards 
the net. Brazil had scored the first of its four goals in the final game of the 1970 World Cup anda 
whole country was about to start dancing in the streets. 


behaviour illustrated here, as well as the 

gift of remembering the multitude of 
sensations associated with an instant of joy 
many decades ago, offer us a glimpse of the 
awesome repertoire of tasks that the human 
brain can accomplish. Through mechanisms 
that still elude our comprehension, the 
electrical activity of millions of brain cells 
(neurons) can be translated into precise 
sequences of skilled movements. Coordinat- 
ed neuronal activity also provides us with 
exquisite perceptual and sensorimotor capa- 
bilities, illustrated in this example by Pelé’s 
ability to track the ball’s trajectory and plan 
the timing of his jump to hit it head on. 
But this is not all. Highly distributed patterns 
of neuronal firing underlie our ability to 
generate expectations about the outcome ofa 
future event, learn the complex laws of nature 
and create art. One could argue, therefore, 
that hidden within the intricate principles 
that govern the way brain circuits operate lies 
the key to understanding the very essence of 
what it is to be human. 

Witnessing the relentless growth of 
the disciplines that define modern neuro- 
science, one cannot help wondering what 
kind of insights, clinical applications and 
technologies may emerge from brain 
research in the future, and, more important, 
how they will impact on our lives. Although 
many of the imagined possibilities may not 
be feasible at this time, recent work indicates 
that some current ideas will come to fruition 
in the not-so-distant future. Here, I focus 
on one of these — the development of 
direct interfaces between machines and 
the human brain. 


iE vast range of human abilities and 
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Brain-machine interfaces 

I propose that the introduction of new meth- 
ods for measuring large-scale brain activity, 
new techniques for microstimulating 
neuronal tissue, and emerging developments 
in microchip design, computer science and 
robotics have the potential to coalesce into a 
new technology devoted to creating inter- 
faces between the human brain and artificial 
devices. One day, it is conceivable that such 
technology could allow patients to use brain 
activity to control electronic, mechanical or 
even virtual devices, leading to new 
therapeutic alternatives for restoring lost 
sensory, motor and even cognitive functions. 
Although many fundamental neurobiological 
questions and technical difficulties need to be 
solved, we can be optimistic about the feasibili- 
ty ofimplementing this concept in the next few 
decades. Indeed, one brain—machine interface 
— the auditory prosthesis known as the 
cochlear implant — was introduced years ago 
and has improved the quality of life of many 
deaf patients” (see Box 1). 

Neuroscientists have long relished the 
possibility of using brain signals to control 
artificial devices’. As a consequence, there 
are already many terms in the literature’ to 
describe devices that could accomplish this 
goal (for example, brain-actuated technolo- 
gy, neuroprostheses or neurorobots). Here I 
will refer to these devices collectively as 
‘hybrid brain—machine interfaces (HBMIs)’. 
The word ‘hybrid’ reflects the fact that these 
applications rely on continuous interactions 
between living brain tissue and artificial 
electronic or mechanical devices. 

My definition of HBMIs incorporates two 
main types of application. Type 1 devices use 
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artificially generated electrical signals to 
stimulate brain tissue in order to transmit 
some particular type of sensory information 
or to mimic a particular neurological func- 
tion. The classic example of this application is 
an auditory prosthesis. Future applications 
aimed at restoring other sensory functions, 
such as vision, by microstimulation of specif- 
ic brain areas would also belong to this group. 
In addition, type 1 HBMIs include methods 
for direct stimulation of the brain to alleviate 
pain, to control motor disorders such as 
Parkinson’s disease’, and to reduce epileptic 
activity by stimulation of cranial nerves®. 
These last three applications rely on the 
observation that direct microstimulation of 
brain tissue can disrupt pathological patterns 
of brain activity that underlie some 
neurological disorders. Type 2 HBMIs rely on 
the real-time sampling and processing of 
large-scale brain activity to control artificial 
devices. An example of this application 
would be the use of neural signals derived 
from the motor cortex to control the 


Cochlear implants: 
the first HBMI 


Auditory prostheses work by converting features of 
acoustic signals, such as speech, into patterns of 
electrical stimuli that are then delivered through 
an array of chronically implanted electrodes to 
auditory nerve fibres lying on the basilar 
membrane of the cochlea. As the basilar 
membrane contains a representation of sound 
frequencies, known as a tonotopic map, auditory 
prostheses deliver high-frequency information to 
the basal region of the cochlea, and low-frequency 
signals to the apical region, to mimic normal 
auditory processing. More than 30,000 deaf 
patients, ranging in age from 12 months to 80 
years, have had such devices successfully 
implanted’. Although results vary from case to 
case, even slight improvements in auditory 
performance have helped people to communicate 
better and to become more aware of their 
surrounding environment. 
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movements ofa prosthetic robotic arm in real 
time. Obviously, clinical applications that 
require reciprocal interaction between the 
brain and artificial devices will combine both 
type 1 and2 HBMIs. 

The design and implementation of 
HBMIs will involve the combined efforts of 
many areas of research, suchas neuroscience, 
computer science, biomedical engineering, 
very large scale integration (VLSI) design 
and robotics. I have selected a few current 
developments in these fields to illustrate 
below some of the conceptual advances and 
technologies that will be required to design 
and implement useful HBMIs. I will then 
describe two potential clinical applications 
of such technology that should emerge in the 
near future: a system to monitor and treat 
epileptic seizures and a device to control a 
robotic prosthetic arm. 


Building a HBMI 

The first of the many challenges associated 
with the development ofany HBMI is the need 
to understand better the principles by which 
neural ensembles encode sensory, motor and 
cognitive information. This is rapidly 
becoming one of the main goals of modern 
neuroscience, but our present knowledge is 
elementary at best. In the case of motor con- 
trol, for instance, the areas of the primate brain 
involved are well known, and considerable 
information is available on the physiological 
properties of individual neurons located in 
each of them. But we knowlittle about how the 
brain makes use of information from these 
neurons to generate movements. To design a 
type 2 HBMI that uses brain-derived signals to 
control a prosthetic robotic arm, we will need 
to learn how to sample and decode the motor 
signals generated by neurons and how to feed 
them into an artificial device to mimic the 
intended movement. 


Recording brain activity 

It is clear that neurobiological principles will 
be central in devising a strategy to overcome 
these hurdles. For example, classic experi- 
ments in primates have demonstrated that 
fundamental parameters of motor control 
emerge by the collective activation of large 
distributed populations of neurons in the pri- 
mary motor cortex (M1). Single M1 
neurons are broadly tuned to the direction of 
force required to generate a reaching arm 
movement’. In other words, even though 
these neurons fire maximally before the 
execution of a movement in one direction, 
they also fire significantly before the onset of 
arm movements in a broad range of other 
directions. Therefore, to compute a precise 
direction of arm movement, the brain may 
have to perform the equivalent of a neuronal 
‘vote’ or, in mathematical terms, a vector sum- 
mation of the activity of these broadly tuned 
neurons’. This implies that to obtain the 
motor signals required to control an artificial 
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Figure 1 Schematic description of the general organization of a type 2 HBMI. 


device we will need to sample the activity of 
many neurons simultaneously and design 
algorithms capable of extracting motor 


control signals from these ensembles. 
Moreover, it will be crucial to investigate how 
these neural ensembles interact under more 
complex and ‘real-world’ experimental con- 
ditions® to generate different motor behav- 
iours. These data will be vital to answering 
basic questions in regard to the development 
of type 2 HBMIs. For example, what is the 
minimum neuronal sample required to gen- 
erate reliable brain-derived control signals? 
Should these samples be obtained from one or 
multiple brain areas? Does the same popula- 
tion of neurons code for single or multiple 
control parameters? Finally, how might 
neural encoding mechanisms change with 
time, experience and learning? 

Figure | illustrates the general organiza- 
tion of a type 2 HBMI and depicts some of 
the technological challenges involved in 
designing such devices. The first design step 
involves the selection ofa technique (Fig. la) 
that yields reliable, stable and long-term 
recordings of brain activity that can be used 
as control signals to drive an artificial device. 
From recent studies in animals””°, clinical 
applications of HBMIs will probably require 
sampling of large numbers of neurons (in the 
order of hundreds or thousands) with a 
temporal resolution of 10-100 ms, depend- 
ing on the application. 

Although neuroscientists have long rec- 
ognized the need to investigate the properties 
of large neural ensembles", it is very difficult 
to obtain reliable, long-term measurements 
of neural ensemble activity with high spatial 
and temporal resolution. Starting in the 
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1940s and 50s with multichannel recordings 
of scalp electroencephalographic (EEG) 
activity and of the general electrical activity 
evoked by movement or sensory stimulation, 
a variety of metabolic, optical and electro- 
physiological methods have been introduced 
for monitoring large-scale brain activity. 
Modern multichannel electrophysiological 
recordings are made from arrays of micro- 
electrodes surgically implanted in the brain. 
They currently allow neurophysiologists to 
record simultaneously, with a resolution of 
milliseconds, the extracellular activity of up 
to 100 individual neurons, distributed across 
multiple brain structures, in animals carry- 
ing out some task or other’. Although future 
improvements might allow long-term and 
non-invasive sampling of human neural 
activity with the same temporal resolution as 
intracranial recordings, the first generation 
of HBMIs will probably rely on improved 
versions of electrophysiological methods, 
such as multichannel EEG or multielectrode 
intracranial recordings. Indeed, preliminary 
studies in paralysed patients have shown that 
EEG signals can be used to trigger the move- 
ment of computer cursors” or offer a way for 
patients to communicate”. 

Unfortunately, less invasive electrophysi- 
ological methods, such as scalp EEG record- 
ings that reflect the common electrical 
activity of millions of neurons in widespread 
areas of the cortex, lack the resolution to 
provide the kind of time-varying motor 
signals needed to control a robotic arm in 
real time*. Multichannel intracranial record- 
ings of brain activity, obtained by surgical 
implantation of arrays of microwires within 
one or more cortical motor areas, will 
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Figure 2 A prototype of an instrumentation neurochip for processing brain-derived signals. This 


chip, containing a portion of the analog signal processing for 16 neural channels, was designed by 
I. Obeid, H. Aurora, J. Morizio and P. Wolf in the Departments of Biomedical and Electrical & 
Computer Engineering at the Pratt School of Engineering, Duke University. The mixed-signal 


CMOS (complementary metal-oxide semiconductor) process used in the design supports digital 


signal processing modules which will be included in future generations of this device. 


therefore be required, with mathematical 
analysis of the extracellular activity of small- 
er populations (100—1,000) of neurons pro- 
viding the raw brain signals for use in most 
HBMIs”’. Despite some degree of recording 
degradation over time, present technology 
allows simultaneous sampling of 50-100 
neurons, distributed across multiple cortical 
areas of small primates, to remain viable for 
several years'”’”, Technological advances in 
multielectrode array design and neural 
signal instrumentation in the next decade 
alone are expected to increase the number of 
neurons that can be recorded simultaneous- 
ly byat least one order of magnitude. 

The precise placement of the electrode 
arrays for intracranial recording may not be 
as critical to the ability to control an artificial 
device as was first conjectured. As motor 
control signals emerge from the distributed 
activation of large populations of neurons, 
and as cortical and subcortical neurons are 
capable of considerable plastic reorganiza- 
tion during adulthood”, electrode arrays 
targeted to brain areas of interest may suffice 
in most cases. As subjects learn to interact 
with artificial devices through HBMIs, it is 
likely that sampled neurons that were not 
originally involved in the type of motor 
control to be mimicked may be recruited 
into generating the signals required to 
control artificial devices. 


Generating the output 

After selecting a method for acquiring the 
necessary brain signals, the next challenge is 
to design the instrumentation (Fig. 1b—d) 
required for recording and processing these 
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signals in real time. Currently, this requires 
specialized, sizeable and expensive electron- 
ic equipment, which can amplify and filter 
the original signal as well as perform analog- 
to-digital conversion to facilitate further 
processing and storage of data. To make 
HBMIs viable, new technologies for 
portable, wireless-based, multichannel 
neural signal instrumentation are needed. 

The central issue of signal conditioning 
and instrumentation may be solved in the near 
future by the application of mixed-signal VLSI 
into the design of neurophysiological instru- 
mentation chips. This technology allows 
analog and digital signals to coexist in the same 
microchip, and has the potential to provide the 
multichannel, programmable and low-noise 
package required for conditioning brain- 
derived signals for clinical implementation of 
HBMIs. Moreover, the resulting microchip 
would be small enough to be chronically 
implanted in patients and could be powered by 
replaceable batteries. Such microchips could 
rely on wireless communication protocols 
based on a radio frequency link to broadcast 
neural signals to other components of the 
HBMI (Fig. 1d-e). 

Prototypes of dedicated ‘instrumenta- 
tion neurochips’ (Fig. 2) are currently being 
developed, although many complex issues 
must be solved before they can become 
operational'®. For instance, efficient solu- 
tions will have to be found to provide enough 
power for performing analog and digital 
processing, while still ensuring that signals 
can be transmitted by telemetry. Thus, 
battery technology, device packing and the 
bandwidth of the neural signals, among 
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other factors, will certainly be important in 
the design of HBMIs"*. 

Having selected a method for sampling and 
conditioning brain signals, the next step —and 
one of the most difficult challenges — is to 
define a strategy for extracting meaningful 
control information from neural ensemble 
activity in real time. Currently, neuroscientists 
rely on a variety of linear and nonlinear 
multivariate algorithms, such as discriminant 
analysis, multiple linear regression and artifi- 
cial neural networks, to carry out real-time and 
off-line analysis of neural ensemble data. 
Preliminary results from animal studies that 
use these different methods are encouraging, 
but considerably more experience is needed to 
apply these techniques in clinical HBMIs. The 
challenge is to produce algorithms that can 
combine the activity of large numbers of 
neurons, which convey different amounts of 
information, and extract stable control signals, 
even when the firing patterns of these neurons 
change significantly across different timescales. 
Research on areas ranging from automatic 
sorting algorithms for unsupervised isolation 
ofsingle neuron action potentials, to the design 
of real-time pattern recognition algorithms 
that can handle data from thousands of simul- 
taneously recorded neurons will certainly be 
required. In the same context, clinical applica- 
tions of HBMIs will require considerable 
computational resources. 

In the not too distant future, new develop- 
ments in the design of brain-inspired VLSI’, 
an exciting area of research aimed at modelling 
neuronal systems in silicon'*, may provide the 
means for achieving the type of efficient real- 
time neural signal analysis required for 
HBMIs. This technology may allow pattern 
recognition algorithms, such as artificial 
neural networks or realistic models of neural 
circuits, to be implemented directly in silicon 
circuits. Among many other technical hurdles, 
significant work will be required to make these 
silicon circuits adaptive, perhaps by incorpo- 
rating learning rules derived from the study of 
biological neural circuits. This will allow 
‘training’ of algorithms as well as ensuring the 
robustness of the control system. From an 
implementation point of view, ‘analytical 
neurochips’ are ideal as they could be inter- 
faced with the instrumentation neurochip and 
be chronically implanted in the subject. 

The final component of the idealized 
HBMI (Fig. le—f) isa real-time control inter- 
face which uses processed brain signals to 
control an artificial device. The types of 
devices used are likely to vary considerably in 
each application, ranging from elaborate 
electrical pattern generators to control mus- 
cles, to complex robotic and computational 
devices designed to augment motor skills”. 


HBMls for epilepsy control 

Estimates indicate that about 0.5—2.0% of the 
population has epilepsy”. About 10-50% of 
these patients do not respond well to current 
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antiepileptic medications and may not be 
candidates for surgery. Throughout this 
century, neuroscientists have used multi- 
channel recording from scalp, brain surface 
and even chronically implanted intracranial 
electrodes to investigate the electrophysio- 
logical activity that characterizes different 
types of seizure in humans. By doing so, 
scientists have not only identified different 
types of epilepsy, but they have also learned 
that there are distinct patterns of neurophysi- 
ological activity associated with the initiation 
and establishment ofa seizure attack. 

Several exciting new developments in 
epilepsy research indicate that the develop- 
ment ofan unsupervised HBMI for monitor- 
ing, detecting and treating seizure activity 
may be possible in the next decade (Fig. 3a). 
First, for certain types of seizure, there seems 
to be a particular spatiotemporal pattern of 
cortical activity that appears seconds or even 
minutes before the full epileptic attack 
starts’. Recently, a few laboratories have 
introduced automatic seizure-prediction 
algorithms that can be applied to intracranial 
and scalp recordings to forecast the occur- 
rence of a seizure’, These and future 
seizure-prediction algorithms might provide 
sufficient time (2-5 minutes) to warn the 
patient of an imminent attack, and to trigger 
automatic therapeutic intervention before 
convulsion or loss of consciousness. 

But what kind of therapy could be 
triggered that would workin patients who are 
refractory to epilepsy medication? The 
answer may lie in another recent develop- 
ment in epilepsy research. Studies in both 
animals” and human subjects® have revealed 
that electrical stimulation of peripheral 
cranial nerves, such as the vagus’ and 
trigeminal” nerves, can substantially reduce 
cortical epileptic activity. Moreover, if this 
peripheral nerve stimulation is applied 
before the initiation of seizure or during its 
initial stages, significantly higher reduction 
of seizure activity can be achieved. 

From this I believe thata device containinga 
combination ofboth type 1 and2 HBMIscould 
be designed to function somewhat like a 
modern heart pacemaker (Fig. 3a). This ‘brain 
pacemaker’ would rely on arrays of chronically 
implanted electrodes to search continuously 
for spatiotemporal patterns of cortical 
activity indicating an imminent epileptic 
attack. Instrumentation neurochips would be 
responsible for all the basic signal-processing 
operations. They would also provide signals to 
one or more seizure-prediction algorithms, 
implemented into analytical neurochips, 
which would carry out real-time analysis of 
cortical activity. Once pre-seizure activity pat- 
terns were detected, the analytical neurochip 
could trigger electrical stimulation of one or 
multiple cranial nerves. In patients who 
respond to pharmacological therapy, the same 
stimulator could be used to activate a mini- 
pump to deliver one or more anti-epileptic 
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Figure 3 Schematic description of two potential applications of type 2 HBMIs. a, Design of a ‘brain 
pacemaker’ that monitors neural activity using a VLSI chip designed to detect seizure activity. 
When seizure activity is detected, the VLSI chip sends a signal to an implanted stimulus generator 
that drives either a nerve cuff electrode or a mini-pump for drug delivery, either of which can stop 
the seizure activity. b, HBMI for controlling a robotic prosthetic arm using brain-derived signals. 


Multiple, chronically implanted, intracranial microelectrode arrays would be used to sample the 
activity of large populations of single cortical neurons simultaneously. The combined activity of 
these neural ensembles would then be transformed by a mathematical algorithm into continuous 
three-dimensional arm-trajectory signals that would be used to control the movements of a 
robotic prosthetic arm. A closed control loop would be established by providing the subject with 
both visual and tactile feedback signals generated by movement of the robotic arm. 


drugs directly into the blood stream. Recently, a 
simplified implementation of this concept has 
been used successfully in rats“, giving hope 
that a brain pacemaker for seizure monitoring 
and controlin humans may not be far ahead. 


HBMls to restore motor function 
Another clinical application of HBMIs that 
could emerge in the near future aims at 
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restoring different aspects of motor function 
in patients with severe body paralysis, caused 
primarily by strokes, spinal cord lesions or 
peripheral degenerative disorders (Fig. 3b). 
Advances in this rapidly growing field of 
research indicate that neural signals from 
healthy regions of the brain could be used to 
control the movements of artificial prosthet- 
ic devices, such asa robotic arm. Preliminary 
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findings also demonstrate that paralysed 
patients can learn to use brain signals 
obtained from their motor cortex to interact 
with computers”. 

Extensive electrophysiological work in 
primates and imaging studies in humans have 
shown that multiple interconnected cortical 
areas in the frontal and parietal lobes are 
involved in the selection of motor commands 
that control the production of voluntary arm 
movements”. Although each of these areas 
has different degrees of functional specializa- 
tion, in theory, each of them could be selected 
as the source of brain signals for controlling 
the movements of an artificial device. Within 
each of these cortical areas, different motor 
parameters, such a force and direction of 
movement, are coded by the distributed 
activity of populations of neurons, each of 
which is typically broadly tuned to one (or 
more) of these parameters. This indicates that 
implementations of HBMIs for robotic arm 
control need to rely on intracranial recordings 
from large populations of single neurons to 
derive motor control signals. 

At a first glance, a random sample of 
100-1,000 cortical motor neurons, which 
represents a reasonable expectation for the 
yield of multielectrode intracranial record- 
ings in the near future, may look too small to 
unveil any useful information. But recent 
neurophysiological experiments dispute this 
view. For instance, currently one can obtain 
precise off-line reconstructions of complex 
three-dimensional arm trajectories by using 
simple multiple regression techniques to 
transform the activity of 300-400 serially 
recorded cortical motor neurons into a 
neural population vector”. Moreover, exper- 
iments in rats’ and primates'® have shown 
that simple, real-time algorithms, applied to 
samples of 50-100 simultaneously recorded 
cortical neurons, can be used to control 
robotic devices in real time and mimic the 
type of three-dimensional arm reaching 
movements produced by primates. 

Another important issue is that, to achieve 
seamless interactions with prosthetic devices, 
patients will have to receive sensory feedback 
information (for example visual or tactile 
signals) from the prosthetic limbs. These 
feedback signals will establish a closed control 
loop between the brain and artificial devices 
and will probably help patients learn how to 
operate HBMIs. Studies in rats have revealed 
that, if subjects receive visual feedback infor- 
mation as they learn to use brain activity to 
interact with a robotic arm, and are rewarded 
for the successful completion of these move- 
ments, they progressively cease to produce 
overt limb movements’. In other words, even 
though the rats continued to exhibit the 
patterns of cortical activity required to control 
the movements of the robotic arm, this motor 
activity did not result in any significant limb 
movement. This indicates that motor control 
signals can be generated by cortical neurons 
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without any muscle activity, and hence that 
paralysed patients might be capable of learn- 
ing to operate a robotic arm even though they 
cannot move their own limbs. 

These observations also raise the intrigu- 
ing hypothesis that, by establishing a closed 
control loop with an artificial device (Fig. 3b), 
the brain could incorporate electronic, 
mechanical or even virtual objects into its 
somatic and motor representations, and 
operate upon them as if they were simple 
extensions of our own bodies. The fact that the 
adult cortexis capable of significant functional 
reorganization (or plasticity) after peripheral 
and central injuries’, changes in sensory 
experience™ and learning ofnew motor skills” 
supports this possibility. Indeed, the notion 
that adult plasticity can dynamically alter the 
perception of the limits ofour own bodyis cor- 
roborated by studies on patients who have 
undergone limb amputations. Immediately 
after the amputation, most of these patients 
experience the sensation that their amputated 
limb is still present and moving. These 
‘phantom limb’ sensations are paralleled by a 
significant plasticity of body maps in the 
somatosensory cortex”, the part of the brain 
that receives and interprets sensory signals 
from areas such as the skin surface. Instead of 
remaining silent, the areas in these brain maps 
that used to represent the amputated limb 
progressively start to respond to stimulation of 
neighbouring body regions spared by the 
amputation. Thus, it is conceivable that tactile 
feedback signals, generated by the movements 
of a brain-controlled robotic arm and 
delivered to the patient’s skin, could be used to 
incorporate the representation of such an 
artificial device into cortical and subcortical 
somatotopic maps. 

Undoubtedly, years of research will be 
required and many fundamental technologi- 
cal breakthroughs needed before this comes 
close to reality. Nevertheless, it seems 
reasonable to predict that a definitive 
demonstration of such a phenomenon could 
trigger a revolution in the way future genera- 
tions interact with computers, virtual 
objects and remote environments, by allow- 
ing never-before-experienced augmentation 
of perceptual, motor and cognitive capabili- 
ties. Such applications, however, will require 
the introduction of new, non-invasive 
methods for sampling brain activity. 


A final thought 

Although developing expectations ofa distant 
future isa risky business and may raise unjusti- 
fied hope that solutions are just around the 
corner, I cannot avoid ending this brief 
overview on an optimistic tone. Despite many 
significant conceptual and_ technological 
obstacles, the possibility of developing clinical 
applications of HBMIs is real and worth 
pursuing, especially given the potential bene- 
fits that they may bring to people afflicted by 
neurological disorders. At the very least, 
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research on HBMIs will yield powerful new 
tools to investigate hypotheses of how large 
populations of neurons process information 
and adapt according to changes in experience. 
Some may argue that one could achieve this 
goal just by building theoretical models and 
running computational simulations. Perhaps 
thatis true. But as my good friend Idan Segev, a 
leading computational neuroscientist, always 
tells me, there is a subtle but fundamental 
difference between simulating reality and 
building it. Those of us who saw Pelé scoring 
that magic goal on that hot Mexican afternoon 
in 1970 and dreamed about doing the same 
thing would certainly agree. 

Miguel A. L. Nicolelis is in the Departments of 
Neurobiology, Experimental Psychology, and 
Biomedical Engineering, Duke University, Durham, 
North Carolina 27710, USA 
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People with nerve or limb injuries may one day be able 


to command wheelchairs, prosthetics and even paralyzed arms 


and legs by “thinking them through” the motions 


inside a soundproof chamber at our Duke University lab- 
oratory. Her right hand grasped a joystick as she watched 
a horizontal series of lights on a display panel. She knew 
that if a light suddenly shone and she moved the joystick 
left or right to correspond to its position, a dispenser would 
send a drop of fruit juice into her mouth. She loved to play 
this game. And she was good at it. 

Belle wore a cap glued to her head. Under it were four 
plastic connectors. The connectors fed arrays of micro- 
wires—each wire finer than the finest sewing thread—into 
different regions of Belle’s motor cortex, the brain tissue 
that plans movements and sends instructions for enacting 
the plans to nerve cells in the spinal cord. Each of the 100 
microwires lay beside a single motor neuron. When a neu- 
ron produced an electrical discharge—an “action poten- 
tial”—the adjacent microwire would capture the current 
and send it up through a small wiring bundle that ran from 
Belle’s cap to a box of electronics on a table next to the 
booth. The box, in turn, was linked to two computers, one 
next door and the other half a country away. 

In a crowded room across the hall, members of our re- 
search team were getting anxious. After months of hard 
work, we were about to test the idea that we could reliably 
translate the raw electrical activity in a living being’s 
brain—Belle’s mere thoughts—into signals that could direct 
the actions of a robot. Unknown to Belle on this spring af- 
ternoon in 2000, we had assembled a multijointed robot 
arm in this room, away from her view, that she would con- 
trol for the first time. As soon as Belle’s brain sensed a lit 


OWL MONKEY named Belle climbs on a robot arm she was able to 
control from a distant room purely by imagining her own arm moving 
through three-dimensional space. 
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Belle, our tiny owl monkey, was seated in her special chair 


spot on the panel, electronics in the box running two real- 
time mathematical models would rapidly analyze the tiny 
action potentials produced by her brain cells. Our lab com- 
puter would convert the electrical patterns into instructions 
that would direct the robot arm. Six hundred miles north, 
in Cambridge, Mass., a different computer would produce 
the same actions in another robot arm, built by Mandayam 
A. Srinivasan, head of the Laboratory for Human and Ma- 
chine Haptics (the Touch Lab) at the Massachusetts Insti- 
tute of Technology. At least, that was the plan. 

If we had done everything correctly, the two robot arms 
would behave as Belle’s arm did, at exactly the same time. 
We would have to translate her neuronal activity into ro- 
bot commands in just 300 milliseconds—the natural delay 
between the time Belle’s motor cortex planned how she 
should move her limb and the moment it sent the instruc- 
tions to her muscles. If the brain of a living creature could 
accurately control two dissimilar robot arms—despite the 
signal noise and transmission delays inherent in our lab net- 
work and the error-prone Internet—perhaps it could some- 
day control a mechanical device or actual limbs in ways 
that would be truly helpful to people. 

Finally the moment came. We randomly switched on 
lights in front of Belle, and she immediately moved her joy- 
stick back and forth to correspond to them. Our robot arm 
moved similarly to Belle’s real arm. So did Srinivasan’s. 
Belle and the robots moved in synchrony, like dancers 
choreographed by the electrical impulses sparking in Belle’s 
mind. Amid the loud celebration that erupted in Durham, 
N.C., and Cambridge, we could not help thinking that this 
was only the beginning of a promising journey. 

In the two years since that day, our labs and several oth- 
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ers have advanced neuroscience, computer science, microelec- 
tronics and robotics to create ways for rats, monkeys and even- 
tually humans to control mechanical and electronic machines 
purely by “thinking through,” or imagining, the motions. Our 
immediate goal is to help a person who has been paralyzed by 
a neurological disorder or spinal cord injury, but whose mo- 
tor cortex is spared, to operate a wheelchair or a robotic limb. 
Someday the research could also help such a patient regain con- 
trol over a natural arm or leg, with the aid of wireless commu- 
nication between implants in the brain and the limb. And it 
could lead to devices that restore or augment other motor, sen- 
sory or cognitive functions. 

The big question is, of course, whether we can make a prac- 
tical, reliable system. Doctors have no means by which to re- 
pair spinal cord breaks or damaged brains. In the distant fu- 
ture, neuroscientists may be able to regenerate injured neurons 


There were caveats, however. Georgopoulos had recorded 
the activity of single neurons one at a time and from only one 
motor area. This approach left unproved the underlying hy- 
pothesis that some kind of coding scheme emerges from the si- 
multaneous activity of many neurons distributed across multi- 
ple cortical areas. Scientists knew that the frontal and parietal 
lobes—in the forward and rear parts of the brain, respective- 
ly—interacted to plan and generate motor commands. But tech- 
nological bottlenecks prevented neurophysiologists from mak- 


Belle’s 600-Mile Reach 


ON THE DAY BELLE first moved a multijointed robot arm with her 


thoughts, she wore a cap glued to her head. Beneath the cap, each 


of four plastic connectors fed an array of fine microwires into her 
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cortex (a). As Belle saw lights shine suddenly and decided to move a 
joystick left or right to correspond to them, the microwires detected 
electrical signals produced by activated neurons in her cortex and 


or program stem cells (those capable of differentiating into var- 
ious cell types) to take their place. But in the near future, brain- 
machine interfaces (BMIs), or neuroprostheses, are a more vi- 


able option for restoring motor function. Success this summer 
with macaque monkeys that completed different tasks than 
those we asked of Belle has gotten us even closer to this goal. 


From Theory to Practice 

RECENT ADVANCES in brain-machine interfaces are ground- 
ed in part on discoveries made about 20 years ago. In the ear- 
ly 1980s Apostolos P. Georgopoulos of Johns Hopkins Uni- 
versity recorded the electrical activity of single motor-cortex 
neurons in macaque monkeys. He found that the nerve cells 
typically reacted most strongly when a monkey moved its arm 
in a certain direction. Yet when the arm moved at an angle 
away from a cell’s preferred direction, the neuron’s activity did- 
n’t cease; it diminished in proportion to the cosine of that an- 
gle. The finding showed that motor neurons were broadly tuned 
to a range of motion and that the brain most likely relied on the 
collective activity of dispersed populations of single neurons to 
generate a motor command. 


a Rats and monkeys whose brains have been wired to a 
computer have successfully controlled levers and robot 
arms by imagining their own limb either pressing a bar 
or manipulating a joystick. 

= These feats have been made possible by advances in 
microwires that can be implanted in the motor cortex and 
by the development of algorithms that translate the 
electrical activity of brain neurons into commands able 
to control mechanical devices. 

a Human trials of sophisticated brain-machine interfaces 
are far off, but the technology could eventually help 
people who have lost an arm to control a robotic 
replacement with their mind or help patients with a spinal 
cord injury regain control of a paralyzed limb. 
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relayed the signals to a “Harvey box” of electronics. 


Implanted 
microwire array 


Belle in 
laboratory 
room in 


Durham, N.C. 


be 
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ing widespread recordings at once. Furthermore, most scien- 
tists believed that by cataloguing the properties of neurons one 
at a time, they could build a comprehensive map of how the 
brain works—as if charting the properties of individual trees 
could unveil the ecological structure of an entire forest! 
Fortunately, not everyone agreed. When the two of us met 
14 years ago at Hahnemann University, we discussed the chal- 
lenge of simultaneously recording many single neurons. By 
1993 technological breakthroughs we had made allowed us to 


The box collected, filtered and amplified the signals and 
relayed them to a server computer in a room next door. The 
signals received by the box can be displayed as a raster plot (b); 
each row represents the activity of a single neuron recorded over 
time, and each color bar indicates that the neuron was firing at a 
given moment. 

The computer, in turn, predicted the trajectory that Belle’s arm 


Computer (/eft} and robot arm 
(right) in room across the hall 
from Belle 


Raster plot 


Server in room 
next door to Belle’s 


cL 
Ethernet 


Harvey box 
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record 48 neurons spread across five structures that form a rat’s 
sensorimotor system—the brain regions that perceive and use 
sensory information to direct movements. 

Crucial to our success back then—and since—were new 
electrode arrays containing Teflon-coated stainless-steel micro- 
wires that could be implanted in an animal’s brain. Neuro- 
physiologists had used standard electrodes that resemble rigid 
needles to record single neurons. These classic electrodes worked 
well but only for a few hours, because cellular compounds col- 


would take (c) and converted that information into commands for 
producing the same motion in a robot arm. Then the computer 
sent commands to a computer that operated a robot armin a room 
across the hall. At the same time, it sent commands from our 
laboratory in Durham, N.C., to another robot in a laboratory 
hundreds of miles away. In response, both robot arms moved in 
synchrony with Belle’s own limb. —M.A.L.N. and J.K.C. 
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lected around the electrodes’ tips and eventually insulated them 
from the current. Furthermore, as the subject’s brain moved 
slightly during normal activity, the stiff pins damaged neurons. 
The microwires we devised in our lab (later produced by 
NBLabs in Denison, Tex.) had blunter tips, about 50 microns 
in diameter, and were much more flexible. Cellular substances 
did not seal off the ends, and the flexibility greatly reduced neu- 
ron damage. These properties enabled us to produce recordings 
for months on end, and having tools for reliable recording al- 
lowed us to begin developing systems for translating brain sig- 
nals into commands that could control a mechanical device. 
With electrical engineer Harvey Wiggins, now president of 
Plexon in Dallas, and with Donald J. Woodward and Samuel 
A. Deadwyler of Wake Forest University School of Medicine, 
we devised a small “Harvey box” of custom electronics, like 
the one next to Belle’s booth. It was the first hardware that 
could properly sample, filter and amplify neural signals from 
many electrodes. Special software allowed us to discriminate 
electrical activity from up to four single neurons per microwire 
by identifying unique features of each cell’s electrical discharge. 


A Rat’s Brain Controls a Lever 

IN OUR NEXT EXPERIMENTS at Hahnemann in the mid- 
1990s, we taught a rat in a cage to control a lever with its mind. 
First we trained it to press a bar with its forelimb. The bar was 
electronically connected to a lever outside the cage. When the 
rat pressed the bar, the outside lever tipped down to a chute and 
delivered a drop of water it could drink. 

We fitted the rat’s head with a small version of the brain- 
machine interface Belle would later use. Every time the rat com- 
manded its forelimb to press the bar, we simultaneously record- 
ed the action potentials produced by 46 neurons. We had pro- 
grammed resistors in a so-called integrator, which weighted 
and processed data from the neurons to generate a single ana- 
log output that predicted very well the trajectory of the rat’s 
forelimb. We linked this integrator to the robot lever’s con- 
troller so that it could command the lever. 

Once the rat had gotten used to pressing the bar for water, 
we disconnected the bar from the lever. The rat pressed the bar, 
but the lever remained still. Frustrated, it began to press the bar 
repeatedly, to no avail. But one time, the lever tipped and de- 
livered the water. The rat didn’t know it, but its 46 neurons had 


MIGUEL A. L. NICOLELIS and JOHN K. CHAPIN have collaborated for 
more than a decade. Nicolelis, a native of Brazil, received his M.D. 
and Ph.D. in neurophysiology from the University of Sao Paulo. Af- 
ter postdoctoral work at Hahnemann University, he joined Duke 
University, where he now co-directs the Center for Neuroengineer- 
ing and is professor of neurobiology, biomedical engineering, and 
psychological and brain sciences. Chapin received his Ph.D. in neu- 
rophysiology from the University of Rochester and has held facul- 
ty positions at the University of Texas and the MCP Hahnemann 
University School of Medicine (now Drexel University College of Med- 
icine). He is currently professor of physiology and pharmacology 
at the State University of New York Downstate Medical Center. 
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expressed the same firing pattern they had in earlier trials when 
the bar still worked. That pattern prompted the integrator to 
put the lever in motion. 

After several hours the rat realized it no longer needed to 
press the bar. If it just looked at the bar and imagined its fore- 
limb pressing it, its neurons could still express the firing pattern 
that our brain-machine interface would interpret as motor com- 
mands to move the lever. Over time, four of six rats succeeded 
in this task. They learned that they had to “think through” the 
motion of pressing the bar. This is not as mystical at it might 
sound; right now you can imagine reaching out to grasp an ob- 
ject near you—without doing so. In similar fashion, a person 
with an injured or severed limb might learn to control a robot 
arm joined to a shoulder. 


A Monkey's Brain Controls a Robot Arm 

WE WERE THRILLED with our rats’ success. It inspired us to 
move forward, to try to reproduce in a robotic limb the three- 
dimensional arm movements made by monkeys—animals with 
brains far more similar to those of humans. As a first step, we 
had to devise technology for predicting how the monkeys in- 
tended to move their natural arms. 

At this time, one of us (Nicolelis) moved to Duke and es- 
tablished a neurophysiology laboratory there. Together we 
built an interface to simultaneously monitor close to 100 neu- 
rons, distributed across the frontal and parietal lobes. We pro- 
ceeded to try it with several owl monkeys. We chose owl mon- 
keys because their motor cortical areas are located on the sur- 
face of their smooth brain, a configuration that minimizes the 
surgical difficulty of implanting microwire arrays. The micro- 
wire arrays allowed us to record the action potentials in each 
creature’s brain for several months. 

In our first experiments, we required owl monkeys, includ- 
ing Belle, to move a joystick left or right after seeing a light ap- 
pear on the left or right side of a video screen. We later sat them 
in a chair facing an opaque barrier. When we lifted the barrier 
they saw a piece of fruit on a tray. The monkeys had to reach 
out and grab the fruit, bring it to their mouth and place their 
hand back down. We measured the position of each monkey’s 
wrist by attaching fiber-optic sensors to it, which defined the 
wrist’s trajectory. 

Further analysis revealed that a simple linear summation of 
the electrical activity of cortical motor neurons predicted very 
well the position of an animal’s hand a few hundred millisec- 
onds ahead of time. This discovery was made by Johan Wess- 
berg of Duke, now at the Gothenburg University in Sweden. 
The main trick was for the computer to continuously combine 
neuronal activity produced as far back in time as one second to 
best predict movements in real time. 

As our scientific work proceeded, we acquired a more ad- 
vanced Harvey box from Plexon. Using it and some custom, 
real-time algorithms, our computer sampled and integrated the 
action potentials every 50 to 100 milliseconds. Software trans- 
lated the output into instructions that could direct the actions of 
a robot arm in three-dimensional space. Only then did we try to 
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BRYAN CHRISTIE DESIGN 


A Vision of the Future 


A BRAIN-MACHINE INTERFACE might someday help a patient 
whose limbs have been paralyzed by a spine injury. Tiny 
arrays of microwires implanted in multiple motor cortex areas 
of the brain would be wired to a neurochip in the skull. As the 
person imagined her paralyzed arm moving in a particular 
way, such as reaching out for food on a table, the chip would 
convert the thoughts into a train of radio-frequency signals 
and send them wirelessly to a small battery-operated 
“backpack” computer hanging from the chair. 

The computer would convert the signals into motor 
commands and dispatch them, again wirelessly, to a different 
chip implanted in the person’s arm. This second chip would 
stimulate nerves needed to move the arm muscles in the 
desired fashion. Alternatively, the backpack computer could 
control the wheelchair’s motor and steering directly, as the 
person envisioned where she wanted the chair to roll. Or the 
computer could send signals to a robotic arm if a natural arm 
were missing or to a robot arm mounted on a chair. Patrick D. 
Wolf of Duke University has built a prototype neurochip and 
backpack, as envisioned here. —M.A.L.N, and J.K.C. 
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use a BMI to control a robotic device. As we watched our multi- 
jointed robot arm accurately mimic Belle’s arm movements on 
that inspiring afternoon in 2000, it was difficult not to ponder 
the implausibility of it all. Only 50 to 100 neurons randomly 
sampled from tens of millions were doing the needed work. 

Later mathematical analyses revealed that the accuracy of 
the robot movements was roughly proportional to the number 
of neurons recorded, but this linear relation began to taper off 
as the number increased. By sampling 100 neurons we could cre- 
ate robot hand trajectories that were about 70 percent similar 
to those the monkeys produced. Further analysis estimated that 
to achieve 95 percent accuracy in the prediction of one-dimen- 
sional hand movements, as few as 500 to 700 neurons would 
suffice, depending on which brain regions we sampled. We are 
now calculating the number of neurons that would be needed 
for highly accurate three-dimensional movements. We suspect 
the total will again be in the hundreds, not thousands. 

These results suggest that within each cortical area, the 
“message” defining a given hand movement is widely dissemi- 
nated. This decentralization is extremely beneficial to the animal: 
in case of injury, the animal can fall back on a huge reservoir of 
redundancy. For us researchers, it means that a BMI neuro- 
prosthesis for severely paralyzed patients may require sampling 
smaller populations of neurons than was once anticipated. 

We continued working with Belle and our other monkeys af- 
ter Belle’s successful experiment. We found that as the animals 
perfected their tasks, the properties of their neurons changed— 
over several days or even within a daily two-hour recording ses- 
sion. The contribution of individual neurons varied over time. 
To cope with this “motor learning,” we added a simple routine 
that enabled our model to reassess periodically the contribution 
of each neuron. Brain cells that ceased to influence the predic- 
tions significantly were dropped from the model, and those that 
became better predictors were added. In essence, we designed a 
way to extract from the brain a neural output for hand trajec- 
tory. This coding, plus our ability to measure neurons reliably 
over time, allowed our BMI to represent Belle’s intended move- 
ments accurately for several months. We could have continued, 
but we had the data we needed. 

It is important to note that the gradual changing of neuronal 
electrical activity helps to give the brain its plasticity. The num- 
ber of action potentials a neuron generates before a given move- 
ment changes as the animal undergoes more experiences. Yet 
the dynamic revision of neuronal properties does not represent 
an impediment for practical BMIs. The beauty of a distributed 
neural output is that it does not rely on a small group of neu- 
rons. If a BMI can maintain viable recordings from hundreds 
to thousands of single neurons for months to years and utilize 
models that can learn, it can handle evolving neurons, neuronal 
death and even degradation in electrode-recording capabilities. 


Exploiting Sensory Feedback 

BELLE PROVED THAT A BMI can work fora primate brain. 
But could we adapt the interface to more complex brains? In 
May 2001 we began studies with three macaque monkeys at 
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Duke. Their brains contain deep furrows and convolutions that 
resemble those of the human brain. 

We employed the same BMI used for Belle, with one fun- 
damental addition: now the monkeys could exploit visual feed- 
back to judge for themselves how well the BMI could mimic 
their hand movements. We let the macaques move a joystick in 
random directions, driving a cursor across a computer screen. 
Suddenly a round target would appear somewhere on the 
screen. To receive a sip of fruit juice, the monkey had to posi- 
tion the cursor quickly inside the target—within 0.5 second— 
by rapidly manipulating the joystick. 

The first macaque to master this task was Aurora, an ele- 
gant female who clearly enjoyed showing off that she could hit 
the target more than 90 percent of the time. For a year, our 
postdoctoral fellows Roy Crist and José Carmena recorded the 
activity of up to 92 neurons in five frontal and parietal areas 
of Aurora’s cortex. 

Once Aurora commanded the game, we started playing a 
trick on her. In about 30 percent of the trials we disabled the 
connection between the joystick and the cursor. To move the 
cursor quickly within the target, Aurora had to rely solely on 
her brain activity, processed by our BMI. After being puzzled, 
Aurora gradually altered her strategy. Although she continued 
to make hand movements, after a few days she learned she could 
control the cursor 100 percent of the time with her brain alone. 
Ina few trials each day during the ensuing weeks Aurora didn’t 
even bother to move her hand; she moved the cursor by just 
thinking about the trajectory it should take. 

That was not all. Because Aurora could see her performance 
on the screen, the BMI made better and better predictions even 
though it was recording the same neurons. Although much 
more analysis is required to understand this result, one expla- 
nation is that the visual feedback helped Aurora to maximize 
the BMI’s reaction to both brain and machine learning. If this 
proves true, visual or other sensory feedback could allow peo- 
ple to improve the performance of their own BMIs. 

We observed another encouraging result. At this writing, 
it has been a year since we implanted the microwires in Auro- 
ra’s brain, and we continue to record 60 to 70 neurons daily. 
This extended success indicates that even in a primate with a 
convoluted brain, our microwire arrays can provide long-term, 
high-quality, multichannel signals. Although this sample is 
down from the original 92 neurons, Aurora’s performance with 
the BMI remains at the highest levels she has achieved. 

We will make Aurora’s tasks more challenging. In May we 
began modifying the BMI to give her tactile feedback for new 
experiments that are now beginning. The BMI will control a 
nearby robot arm fitted with a gripper that simulates a grasp- 
ing hand. Force sensors will indicate when the gripper encoun- 
ters an object and how much force is required to hold it. Tac- 
tile feedback—is the object heavy or light, slick or sticky?—will 
be delivered to a patch on Aurora’s skin embedded with small 
vibrators. Variations in the vibration frequencies should help 
Aurora figure out how much force the robot arm should apply 
to, say, pick up a piece of fruit, and to hold it as the robot brings 


52. SCIENTIFIC AMERICAN 


it back to her. This experiment might give us the most concrete 
evidence yet that a person suffering from severe paralysis could 
regain basic arm movements through an implant in the brain 
that communicated over wires, or wirelessly, with signal gen- 
erators embedded in a limb. 

If visual and tactile sensations mimic the information that 
usually flows between Aurora’s own arm and brain, long-term 
interaction with a BMI could possibly stimulate her brain to in- 
corporate the robot into its representations of her body—schema 
known to exist in most brain regions. In other words, Aurora’s 
brain might represent this artificial device as another part of her 
body. Neuronal tissue in her brain might even dedicate itself to 
operating the robot arm and interpreting its feedback. 


Stopping Seizures 


RECENT EXPERIMENTS SUGGEST that brain-machine 
interfaces could one day help prevent brain seizures in 
people who suffer from severe chronic epilepsy, which 
causes dozens of seizures a day. The condition ruins a 
patient’s quality of life and can lead to permanent brain 
damage. To make matters worse, patients usually become 
unresponsive to traditional drug therapy. 

A BMI for seizure control would function somewhat like a 
heart pacemaker. It would continuously monitor the brain’s 
electrical activity for patterns that indicate an imminent 
attack. If the BMI sensed such a pattern, it would deliver an 
electrical stimulus to the brain or a peripheral nerve that 
would quench the 
rising storm or trigger 
the release of 


antiepileptic 
medication. 

At Duke we 
demonstrated the 
feasibility of this 
concept in 


collaboration with 
Erika E. Fanselow, 
now at Brown 
University, and 
Ashlan P. Reid, now 
at the University of 
Pennsylvania. We 
implanted a BMI with arrays of microwires in rats given PTZ, a 
drug that induces repetitive mild epilepsy. When a seizure 
starts, cortical neurons begin firing together in highly 


PET SCAN taken during an epileptic 
seizure highlights regions of excessive 
brain activity in yellow. 


synchronized bursts. When the “brain pacemaker” detected 
this pattern, it triggered the electrical stimulation of the 
large trigeminal cranial nerve. The brief stimulus disrupted 
the epileptic activity quickly and efficiently, without 
damaging the nerve, and reduced the occurrence and 
duration of seizures. —M.A.L.N. and J.K.C. 
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To test whether this hypothesis has merit, we plan to con- 
duct experiments like those done with Aurora, except that an 
animal’s arm will be temporarily anesthetized, thereby remov- 
ing any natural feedback information. We predict that after a 
transition period, the primate will be able to interact with the 
BMI just fine. If the animal’s brain does meld the robot arm into 
its body representations, it is reasonable to expect that a para- 
plegic’s brain would do the same, rededicating neurons that 
once served a natural limb to the operation of an artificial one. 

Each advance shows how plastic the brain is. Yet there will 
always be limits. It is unlikely, for example, that a stroke vic- 
tim could gain full control over a robot limb. Stroke damage is 
usually widespread and involves so much of the brain’s white 
matter—the fibers that allow brain regions to communicate— 
that the destruction overwhelms the brain’s plastic capabilities. 
This is why stroke victims who lose control of uninjured limbs 
rarely regain it. 


Reality Check 

GOOD NEWS NOTWITHSTANDING, we researchers must 
be very cautious about offering false hope to people with seri- 
ous disabilities. We must still overcome many hurdles before 
BMIs can be considered safe, reliable and efficient therapeutic 
options. We have to demonstrate in clinical trials that a pro- 
posed BMI will offer much greater well-being while posing no 
risk of added neurological damage. 

Surgical implantation of electrode arrays will always be of 
medical concern, for instance. Investigators need to evaluate 
whether highly dense microwire arrays can provide viable 
recordings without causing tissue damage or infection in hu- 
mans. Progress toward dense arrays is already under way. Duke 
electronics technician Gary Lehew has designed ways to increase 
significantly the number of microwires mounted in an array that 
is light and easy to implant. We can now implant multiple ar- 
rays, each of which has up to 160 microwires and measures five 
by eight millimeters, smaller than a pinky fingernail. We recently 
implanted 704 microwires across eight cortical areas in a 
macaque and recorded 318 neurons simultaneously. 

In addition, considerable miniaturization of electronics and 
batteries must occur. We have begun collaborating with José 
Carlos Principe of the University of Florida to craft implantable 
microelectronics that will embed in hardware the neuronal 
pattern recognition we now do with software, thereby eventu- 
ally freeing the BMI from a computer. These microchips will 
thus have to send wireless control data to robotic actuators. 
Working with Patrick D. Wolf’s lab at Duke, we have built the 
first wireless “neurochip” and beta-tested it with Aurora. See- 
ing streams of neural activity flash on a laptop many meters 
away from Aurora—broadcast via the first wireless connection 
between a primate’s brain and a computer—was a delight. 

More and more scientists are embracing the vision that BMIs 
can help people in need. In the past year, several traditional neu- 
rological laboratories have begun to pursue neuroprosthetic de- 
vices. Preliminary results from Arizona State University, Brown 
University and the California Institute of Technology have re- 
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cently appeared. Some of the studies provide independent con- 
firmation of the rat and monkey studies we have done. Re- 
searchers at Arizona State basically reproduced our 3-D ap- 
proach in owl monkeys and showed that it can work in rhesus 
monkeys too. Scientists at Brown enabled a rhesus macaque 
monkey to move a cursor around a computer screen. Both groups 
recorded 10 to 20 neurons or so per animal. Their success fur- 
ther demonstrates that this new field is progressing nicely. 

The most useful BMIs will exploit hundreds to a few thou- 
sand single neurons distributed over multiple motor regions in 
the frontal and parietal lobes. Those that record only a small 
number of neurons (say, 30 or fewer) from a single cortical area 
would never provide clinical help, because they would lack the 
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in neuronal responsiveness. The other extreme—recording mil- 
lions of neurons using large electrodes—would most likely not 
work either, because it might be too invasive. 

Noninvasive methods, though promising for some thera- 
pies, will probably be of limited use for controlling prostheses 
with thoughts. Scalp recording, called electroencephalography 
(EEG), is a noninvasive technique that can drive a different kind 
of brain-machine interface, however. Niels Birbaumer of the 
University of Tubingen in Germany has successfully used EEG 
recordings and a computer interface to help patients paralyzed 
by severe neurological disorders learn how to modulate their 
EEG activity to select letters on a computer screen, so they can 
write messages. The process is time-consuming but offers the 
only way for these people to communicate with the world. Yet 
EEG signals cannot be used directly for limb prostheses, be- 
cause they depict the average electrical activity of broad popu- 
lations of neurons; it is difficult to extract from them the fine 
variations needed to encode precise arm and hand movements. 

Despite the remaining hurdles, we have plenty of reasons to 
be optimistic. Although it may be a decade before we witness 
the operation of the first human neuroprosthesis, all the amaz- 
ing possibilities crossed our minds that afternoon in Durham 
as we watched the activity of Belle’s neurons flashing on a com- 
puter monitor. We will always remember our sense of awe as 
we eavesdropped on the processes by which the primate brain 
generates a thought. Belle’s thought to receive her juice was a 
simple one, but a thought it was, and it commanded the out- 
side world to achieve her very real goal. 
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OPINION 


Brain- machine interfaces to 
restore motor function and probe 


neural circuits 


Miguel A.L. Nicolelis 


Recent studies have shown that it is possible 
to create functional, bidirectional, real-time 
interfaces between living brain tissue and 
artificial devices. It is reasonable to predict 
that further research on brain-machine 
interfaces will lead to the development of a 
new generation of neuroprosthetic devices 
aimed at restoring motor functions in severely 
paralysed patients. In addition, | propose that 
such interfaces can become the core of a 
new experimental approach with which to 
investigate the operation of neural systems in 
behaving animals. 


Paralysis, one of the most common and debil- 
itating outcomes of severe damage to the cen- 
tral nervous system, continues to cast along 
shadow of hopelessness on millions of lives 
worldwide. Despite the significant strides 
made by basic and clinical research, few (if 
any) therapeutic options are available at 
present for restoring voluntary motor control 
of thelimbsin patients suffering from exten- 
sive traumatic or degenerative lesions of the 
motor system. 

The prevalence of severe body paralysis is 
high, particularly among young adults. For 
instance, among the leading causes of perma- 
nent paralysis, traumatic spinal cord injuries 
— produced by traffic accidents, acts of vio- 
lence or falls— account for nearly 11,000 new 
cases each year in the United States alone’. In 
all, more than 200,000 patientsin the United 
States live with the motor sequelae of similar 
injuries'. About half of these patients are 


quadriplegic, which means that, owing to 
injury to the cervical spinal cord, they cannot 
move any of their limbsor any other muscle 
below the neck. Quadriplegics depend on 
continuous assistance to accomplish even the 
simplest of motor acts. Whereas most of us 
take for granted our ability to breathe, eat and 
drink, a quadriplegic patient usually cannot 
do any of these without the assistance of 
a machine (such as a ventilator) or a carer. 
For this reason, restoring even the smallest 
of motor skillsin these patients can havea 
profound effect on their quality of life. 

Several new experimental approaches to 
the restoration of motor function lost as 
a result of spinal cord injuries have been 
proposed. M ost focus on ways to repair the 
damaged axons that normally mediate 
communication between cortical (and sub- 
cortical) motor neurons and pools of inter- 
neurons and a@-motor neurons in the grey 
matter of the spinal cord. Despite promising 
results, this approach faces large challenges 
given the difficulty involved in guiding large 
numbers of severed axons to re-establish their 
original connections. 

Parallel to these efforts to repair spinal cord 
connectivity, recent experimental demonstra- 
tionsin rodents*’, primates*’ and patients® # 
have raised interest in the proposition that 
neuroprosthetic devices— designed to bypass 
spinal cord lesions— could beused to restore 
basic motor functions in patients suffering 
from severe body paralysis. This approach, 
first proposed by the neurophysiologist 


Edward Schmidt”, assumes that voluntary 
motor commands can be extracted in real 
timefrom the collective electrical activity of 
populations of cortical or subcortical neurons 
spared by the underlying illness, and then 
used to enact motor function either by 
directly stimulating the patient’s musculature 
or by controlling the movements of artificial 
actuators, such as robot arms. FIGURE 1 shows 
the general design of a cortical neuropros- 
thetic device aimed at restoring upper limb 
movements in quadriplegic patients. This 
device relies on chronic intracranial record- 
ings to obtain large-scale brain activity from 
motor areasin the cortex.A real-time mathe 
matical model isresponsiblefor extracting a 
few motor control commands from the raw 
electrical brain activity. An artificial actuator 
(in this case, a multi-joint robot arm), con- 
trolled by theoutput of the real-timemathe 
matical model, is used to reproduce the 
subject's upper limb movements. Because 
the robot arm provides continuous feedback 
information to the subject (not shown), this 
neuroprosthetic device operates through 
aclosed control loop. Recent review articles 
discussthis design in greater detail? 

Thedeviceshown in Fic.1is also defined 
by theneurophysiological approach chosen to 
extract the raw brain activity from which 
themotor control signals are derived. Over 
the years, distinct sources of neuronal signals, 
ranging from scalp electroencephalograms 
(EEGs)®° to intracranial single unit record- 
ings**3, have been proposed as potential 
sources of control signals to drive various 
neuroprostheses. The advantages and dis- 
advantages of each of these and other neuro- 
physiological approaches have been reviewed. 
Regardless of thetype of brain signal selected, 
several studies have now indicated the feasibility 
of building functional interfaces between 
living brain tissuein experimental animals 
and various electronic, mechanical and com- 
putational devices*”. To alimited degree, clin- 
ical applications of this approach, notably 
those based on EEG recordings, have also 
been implenented??, 
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The recent increase in interest in this 
field of research — commonly referred to as 
neuroengineering — has been driven by the 
expectation that various powerful clinical 
implementations of direct brain- machine 
interfaces (BM Is) might emergein the near 
future. Support for this contention is pro- 
vided by the tangible success of a variety of 
implantable brain stimulators, such as 
cochlear implants for restoring auditory 
function, deep brain stimulators for pain 
management and control of motor disorders 
(such as Parkinson’s disease), and vagal nerve 
stimulators for treating chronic epilepsy. 
As more patients have benefited from this 
approach, the interest in brain stimulation 
technology has grown significantly. Indeed, 
investments in a new generation of these 
devices are rapidly fuelling the emergence of 
an incipient brain-based biomedical industry. 
In essence, this processis similar to the phe- 
nomenon that led to the translation of basic 
science discoveries into revolutionary clinical 
applicationsin the field of cardiology. That 
translational process helped to create the 
backbone of aheart- based biomedical engi- 
neering industry, which today generates bil- 
lions of dollarsin revenue. At present, it is 
not possible to predict the economic impact 
that a brain-based industry will havein the 
future. However, judging by present trends, 
it seems fair to say that brain actuators 
will have a prominent role in the further 
development of thisindustry. 

Although these arguments might 
encourage some neurophysiologists to 
become entrepreneurs, a word of caution is 
justified. M ost BM Is tested in experimental 
animals are not yet ready to be translated 
into practical clinical applications. Much 
basic research must be doneto ensure that 
this approach is safe and can provide suffi- 
cient benefits to justify the type of surgical 
intervention that might be required 
for BMIs to become fully operational. 
Furthermore, significant engineering bottle 
necks have yet to be overcome’. These 
include substantial work in the areas of 
microelectrode array design and testing, 
biocompatibility of chronic brain implants, 
microelectronics (for example, miniaturiza- 
tion of hardware for multichannel neural 
signal conditioning and telemetry), power 
management, real-time computational 
modelling and robotics (a new generation 
of actuators and sensors). These engineering 
developments are prerequisites for moving 
the field from experimental demonstrations 
to clinical implementations that can achieve 
the therapeutic benefit envisioned for 
such devices. 


— 


A 


) 


Figure 1 | Schematic representation of a 
cortical neuroprosthetic device. In the general 
design shown, intracranial recordings are used to 
sample the extracellular activity of a few hundred 
neurons in frontal and parietal cortical areas that 
are involved in the planning of arm and hand 
movements. The combined activity of these 
neuronal populations is processed, in real time, by 
a series of simple mathematical models designed 
to extract motor-control parameters from the raw 
brain signals. The outputs of these models are 
used to control the movements of a robot arm 
that has been designed to allow the patient to 
enact fundamental upper limb movements. 


BMIs and fundamental research 

Because of thar potential clinical rdevance, the 
potential contribution of BM Isto basic brain 
research is often neglected. For example, recent 
findings indicate that BMIs might lead to 
the definition of various new experimental 
models, aimed at investigating the real-time 
Operation of neural circuits in behaving 
animals’. The continuing refinement of 
dectrophysiological, computational and engi- 
neering methods for establishing functional 
interfaces betwen living brain tissue and artifi- 
cial devices has the potential to influence 
experimental moddsin several other fields of 
inquiry, such as cellular, computational and 
cognitive neuroscience*>°, A few examples of 
what the future might bring are already present 


in the literature. These include the use of 
experimental BM Isto investigatehow different 
populations of neuronsin aneural circuit con- 
tribute to the encoding of motor parame 
ters>!”18 and an in vitro implementation of 
hybrid biological and artificial networks to 
study the cellular properties of complex neural 
circuits'®. In addition, recent findings show that 
aBMI could beused to optimizethe operant 
training of experimental animals’. 

These examples illustrate anew approach 
to investigation of the brain, which | call 
real-time neurophysiology». The uniqueness 
of this approach is that theories of brain func- 
tion can be tested under the demanding 
constraints imposed by the need to perform 
efficiently in real time or at thesametimescale 
as the animal's behaviour. An example of this 
approach is shown in Fic. 2, which illustrates a 
new experimental platform that is being used 
in our laboratory to address questionson both 
neural population coding and neuroprosthetic 
design. In thissystem, monkeys learn to pro- 
ducecomplex hand movements in responseto 
arbitrary sensory cues. Chronically implanted 
microwire arrays areused to simultaneously 
record theextracellular activity of hundreds 
of single neurons distributed across several 
cortical and subcortical structures. Using this 
neurophysiological approach, the dynamics 
of several neural circuits can be measured 
simultaneously. M oreover, because neuronal 
recordingsremain stablefor long periods’, this 
approach also allows oneto quantify the physi- 
ological changes that take place in different 
components of aneural circuit as animals learn 
various sensorimotor and cognitive tasks, 

In this type of experimental apparatus, 
several real-time modes can be used to extract 
various motor-control parameters— such as 
direction of hand movement, gripping force, 
hand velocity, acceleration, three- dimensional 
position and so on — from the parallel streams 
of neuronal activity that arerecorded asthe 
animal executes various arm movements. The 
outputs of these models are then used to con- 
trol themovements of amulti-jointed robot 
arm, so that one can investigate the type of 
real-time computation that is required to 
reproducethe animal’sarm movementsin a 
robot arm. Although successful replication of 
theanimal’s arm movementsin a robot does 
not imply that the motor system worksin the 
way proposed by the real-time mode tested in 
these studies, this experimental apparatus can 
bea useful tool in ongoing efforts to dissociate 
motor variables through behavioural training. 
| envisagethat this can be achieved by modify- 
ing thestrategy used to closethe control loop 
between the robot and the animal (see later 
discussion). 
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Figure 2 | Experimental design used to test a closed-loop control brain- machine interface for 
motor control in macaque monkeys. Chronically implanted microwire arrays are used to sample the 
extracellular activity of populations of neurons in several cortical motor regions. Linear and nonlinear real- 
time models are used to extract various motor-control signals from the raw brain activity. The outputs of 
these models are used to control the movements of a robot arm. For instance, while one model might 
provide a velocity signal to move the robot arm, another model, running in parallel, might extract a force 
signal that can be used to allow a robot gripper to hold an object during an arm movement. Artificial visual 
and tactile feedback signals are used to inform the animal about the performance of a robot arm 
controlled by brain-derived signals. Visual feedback is provided by using a moving cursor on a video 
screen to inform the animal about the position of the robot arm in space. Artificial tactile and 
proprioceptive feedback is delivered by a series of small vibromechanical elements attached to the 
animal's arm. This haptic display is used to inform the animal about the performance of the robot arm 
gripper (whether the gripper has encountered an object in space, or whether the gripper is applying 
enough force to hold a particular object). ANN, artificial neural network; LAN, local area network. 


To complete a closed-loop control BMI, 
information describing the performance of 
the robot arm is relayed back to the animal, 
using artificially generated visual and ‘pro- 
prioceptive/tactile’ feedback signals. Visual 
feedback is delivered on amonitor in front 
of the animal, and proprioceptive/tactile 
information can berelayed through an array 
of vibrotactile elements attached to the ani- 
mal’s contralateral arm. As microelectrode 
arrays are implanted in neural circuits that 
process sensory feedback generated by arm 
movements, intracranial microstimulation 
can be used to deliver to the animal feedback 
information that describes the robot's 
performance (not shown). 

When aBM | isin aclosed-loop configura- 
tion, the experimenter can manipulate the 
feedback information that the animal receives 
in several ways. These manipulations can be 
designed to permit neuroscientists to address 
key questions regarding the dynamics of 
sensory and motor information encoding by 
distinct populations of neurons. For example, 
does learning allow a population of neurons 


that normally contributes littleto the encoding 
of agiven movement parameter to enhance 
its contribution? This question could be 
addressed by first comparing the performance 
of populations of neurons from distinct corti- 
cal areas in controlling the movements of a 
robot arm. Then, by selecting only neurons 
from agiven cortical area to feed into the real- 
timemodel, onecould measure whether the 
contribution of these neurons can be enhanced 
by visual and/or tactile feedback, which are 
used to indicatethe error between the move 
ments of theanimal’sarm and therobot arm. 
This apparatus could also be used to 
measure how fast changes in the kinematic 
properties of the motor actuator (robot arm) 
or in thetask contingency affect the encoding 
of motor information by distinct populations 
of neurons!’. For example, suppose that 
animals are rewarded for using their brain 
activity to make a robot arm move in the 
same direction as their own hand. By intro- 
ducing arotation in the output of the model 
that converts the animal's brain activity into 
robot arm movements, and using visual and 
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tactile feedback to illustrate the mismatch 
between the robot arm and the animal’s hand 
movements, onecan test whether populations 
of neurons that encode information about 
hand movements adapt to account for the 
transformation of themodd’s output. Results 
obtained at the single neuron level indicate 
that this would occur!*22, 

Theuse of sensory feedback or reward in 
Operant conditioning of neural activity hasa 
longtradition in neuroscience’. Indeed, the 
concept of building aBM| to restore motor 
function was heavily influenced by the early 
studies of Fetz and colleagues”®2+34, T hese 
landmark experiments showed that monkeys 
could learn to increase the firing rate of indi- 
vidual cortical neuronsif they were provided 
with visual or auditory feedback — which sig- 
nalled the unit firing rate— combined witha 
food reward for attaining high rates. This 
conditioning occurred over afew training ses- 
sions so that when overtraining was achieved, 
monkeys could readily increase thefiring rates 
of newly isolated cortical neurons to levels 
50-500% higher than their normal rates”. 
Fetz also showed that removal of feedback 
and reward led to the return of normal firing 
rate levels? 

The experimental apparatus illustrated 
in FIG. 2 also makes it possible to test the 
hypothesis that, with adequate visual and 
proprioceptive/tactile feedback, animals could 
not only learn to operate a robot arm effi- 
ciently, but they could also incorporate such 
an artificial device into the body represen- 
tations that are present in motor and 
somatosensory cortical and subcortical struc- 
tures!3256, T heoretically, such an assimilation 
could occur as aresult of experience-depen- 
dent plastic reorganization, and could lead to 
significant improvement in the operation of a 
neuroprosthetic device. Such ademonstration 
would not only have considerable clinical 
relevance, but would also raise intriguing 
neurobiological and robotic questions”. 


BMls and distributed neural coding 

The examples described above indicate that 
BM Iscould becomeuseful tools for studying 
the dynamic and distributed nature of neural 
population codingin behaving animals. The 
concept of distributed coding has a long 
history in neuroscience. Theinitial formula- 
tion of thistheoretical model — which guides 
most of the present thinking behind attempts 
to devdop aneuroprosthetic devicefor restor- 
ing motor function — dates back to the work 
of the eclectic nineteenth-century English 
physician and Cambridge physicist, Thomas 
Young’. Young is perhaps best known for 
the double slit experiment, which led to the 
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Figure 3 | Distributed neural coding in colour vision. In 1802, Thomas Young (left) introduced the 
concept of distributed neural coding in his classic trichromatic theory of colour vision. In his formulation, 
the combined response profile of only three retinal receptors (middle), tuned to respond to a broad 
spectrum of light wavelength (right), can account for the representation of any colour in the visible 


spectrum. P, Q, R, S, colour stimuli. 


proposition of the principle of light interfer- 
ence for his fundamental contributions to the 
theory of elasticity of materials, and for his 
efforts to decipher Egyptian hieroglyphics. His 
sole contribution to neuroscience was of equal 
statureto his other intellectual adventures. In a 
paper published in 1802 (ReEF.28), Young pro- 
posed thetrichromatic theory of colour vision 
(FIG.3). With no anatomical or functional evi- 
dence, Young proposed that the combined 
action of just three classes of light receptor in 
the retina (later known as cones) could 
account for the complete spectrum of colour 
sensation experienced by humans. Young indi- 
cated (FIG. 3) that although these receptors 
could bespecialized to respond maximally to 
the presence of oneof thethreemain colours 
(red, blueand yellow), each would also be able 
to respond, albeit less strongly, to light of 
different wavelengths. In other words, each 
receptor would be broadly tuned to a large 
wavelength spectrum. 

Young's formulation predicted that the 
collective or distributed response pattern of 
these three retinal receptors could be used to 
represent the wavelength (or colour) of any 
light stimulusin the visual spectrum unam- 
biguously. Despite lacking any insight into the 
structure of the retina or the brain, Young’s 
ingenious formulation gave rise to the 
concept of distributed neural coding. In this 
scheme, the electrical activity of large and 
spatially distributed populations of neurons 
— rather than singlecells— isresponsiblefor 
representing the attributes of incoming 
sensory stimuli, or for generating the motor 
commands required for the production of a 
voluntary act. 

After Young, many neuroscientists con- 
tributed to the elaboration of the concept of 
distributed neural coding. Perhapsthe most 
influential was Donald H ebb. In his classic 
book T he Organization of Behavior’, pub- 
lished in 1949, Hebb provided a cellular 


analogue to Young's formulation by describ- 
ing the brain entity that, accordingto him, 
would be responsible for the‘grunt’ work of 
computing, storing and representing infor- 
mation in thecentral nervous system. H ebb 
proposed that these functions would be 
carried out by: 


“the cell assembly, a diffuse structure 
comprising [brain] cellsin the cortex and 
diencephalon, capable of acting briefly asa 
closed system, delivering facilitation to other 
such systems...” 


The main reason that one can seriously 
consider using neuroprosthetic devices to 
restore motor function in paralysed patients is 
that motor information is widely distributed 
in populations of neurons in the primary 
motor cortex*° and other motor cortical 
areas>”8, This widespread dispersion of 
information within and between cortical 
areas might explain why random samples of 
relatively small populations of singleneurons 
can provide enough information to re 
construct continuous three-dimensional 
hand trajectories produced by monkeys that 
were trained in simplemotor tasks>’. 

Further experimental evidence indicates 
that cortical and subcortical neural ensembles 


“« preliminary results in our 
laboratory indicate that by 
sampling afew hundred 
neurons simultaneously and 
differentially waghting their 
contribution, onecan extract 
multiple control signals from 
thesame population of 
recorded neurons.” 


can simultaneously represent multiple motor 
and sensory parameters**33, Although thereis 
debate on how this multiplexing is achieved, 
preliminary results in our laboratory indicate 
that by sampling a few hundred neurons 
simultaneously and differentially weighting 
their contribution, onecan extract multiple 
control signals from thesame population of 
recorded neurons. Further studies using BM Is 
in experimental animals could help to answer 
this and other fundamental questions in 
neural ensemble physiology. 


Converging on applications 

Because BM Is offer a new way to study dis- 
tributed neural processing, and because 
addressing some of these basic questions 
might influence the design of future neuro- 
prosthetic devices, research on BM Is offersa 
unique opportunity for promoting conver- 
gent areas of investigation for both basic and 
applied neuroscientists. For example, there is 
much discussion in the literature regarding 
how many neurons need to besampled to 
build aclinically viable BMI for restoring 
upper limb movements. Although the origi- 
nal studies suggested that a few hundred 
neurons could providean ideal sample for 
driving such aBM |5, a couple of laboratories 
have used small neuronal samples (8-30 
neurons) to drive experimental BM Is with 
some success®’, 

Clearly, this question is as pertinent to 
neuroscientists that are interested in how 
motor information is encoded in the brain as 
itis to biomedical engineers that are inter- 
ested in designing and implementing aclini- 
cally relevant BM1.An appropriate answer 
requires the analysis of anumber of factors. 
Three main arguments challengethe notion 
that small samples of neurons could be used 
to driveaclinically relevant BM I. First, to be 
considered as a viabletherapeutic alternative, 
BM Is will have to produce a significant 
improvement in the patient’s quality of life. 
Before now, studies based on reduced samples 
of neurons haveled to limited experimental 
demonstrations of motor control®”°. These 
include using cortical neuronal activity to 
control the movements of a computer cursor? 
or to reproducethedirection and trajectory of 
hand movements for brief periods of time®’. 
A recent study reported that monkeys operat- 
ingaclosed-loop control BMI driven by the 
combined activity of about 18 cortical neurons 
correctly completed 50% or fewer of the task 
trials’. As control of computer cursors can be 
achieved with non-invasive methods, such as 
EEG recordings and electromyogram activ- 
ity®°, proponents of BM Is based on small sam- 
ples of neurons would have to demonstrate 


291 


420 | MAY 2003 | VOLUME 4 


www.nature.com/reviews/neuro 


much more elaborate and reliable levels of 
motor control to justify subjecting patients to 
theneurosurgical procedure that is required 
to render these neuroprostheses functional. 

A second argument against the potential 
clinical relevance of these BM Isis that any 
small reduction in the original population of 
recorded cells could impedethe reliable func- 
tion of the neuroprostheses. For instance, any 
minor postsurgical disruption in electrode 
properties, leading to an inability to record 
from the full original population of neurons, 
could render thistype of BM useless. Natural 
loss or death of just a couple of neurons could 
produce a similarly catastrophic effect. Indeed, 
thenormal time-dependent reduction in neu- 
ronal yield that characterizes some methods 
for chronic multi-electrode recordings would 
lead to the sameoutcome. 

Third, relatively small changes in the physi- 
ological properties of these small samples of 
neurons (such as changesin tuning properties) 
could also reduce the effectiveness of such 
BM Is. Although adaptivelearning algorithms 
can be used to counteract these changes*”, vari- 
ationsin patterns of neuronal firing due to 
changesin attention or arousal can prove diffi- 
cult to handlein real time, particularly if the 
sample of cells used to derive aneural popula- 
tion signal is very small. It could be argued that 
patient training using visual, auditory and tac- 
tile feedback signals might enhancetheinfor- 
mation content of individual neuronsor small 
populations of cells. Even though sensory feed- 
back will probably improvethe paformance of 
BM Isand reduce the overall neuronal sample 
required to operate them, thecrucial denon- 
stration that BM Is based on small populations 
of neuronscan maintain high performance for 
months or years, despite losing a few neurons, 
is still lacking. As BMIs based on brain 
implants would haveto maintain a high leva 
of daily performance for many years, this 
important drawback alone almost certainly 
limits the clinical application of adesign based 
on asmall sampleof neurons. 


Neural coding theory and BMI design 

Ultimately, | believe that the design of a suc- 
cessful BM | for restoring control of upper limb 
movements will haveto take into account gen- 
eral physiological principles of how motor 
signals underlying these movements are 
encoded in the primate brain. Moreover, 
instead of aiming solely to restoremotor func- 
tions by controlling a computer cursor, these 
BM Is must be able to restore fundamental 
hand or arm movements by using either 
the patient's limbs (the most difficult goal) or 
artificial devices— such asrobot arms and 
specially designed exoskeletons*# — as their 


motor actuators. Including a gripper in 
these artificial actuators would also be 
essential. Although these requirements 
make it more difficult to build these 
devices, the successful implementation of 
such aBMI would lead to significant bene- 
fits for severely paralysed patients, while 
providing undisputed clinical justification 
for the need for surgical intervention. 


“Ultimately, | believe that 
the design of a successful 
BM | for restoring control of 
upper limb movements will 
have to take into account 
general physiological 
principles of how motor 
signals underlying these 
movements are encoded in 
the primate brain.” 


Recently, | proposed a neuroprosthetic 
design that could overcomethethree problems 
identified in thediscussion of BM Is based on 
small neuronal samples. This design was 
based on five‘physiologically inspired’ princi- 
ples. Thefirst principle proposes that motor 
information related to hand movementsis rep- 
resented in a distributed way in several cortical 
and subcortical structures that define the 
motor system. Thesecond principle purports 
that, within each of these areas, multiple motor 
parameters (position, velocity, force, direction 
and so on) can be extracted in real timefrom 
the electrical activity of populations of neu- 
rons. This principle assumesthat multiplexing 
of information by neural ensembles is a ubiq- 
uitous property of motor cortical areasin the 
frontal and parietal lobes. The third principle 
contends that to reoroducea given hand trajec- 
tory in arobot arm, onemight need to sample 
from asmall fraction (afew hundred) of all the 
neurons (several million) in each cortical 
motor area(and across the entire motor sys- 
tem) that modulatetherr firing rate beforethe 
onset of ahand movement. As mentioned 
above, several independent empirical observa- 
tions support this contention. Thesefindings 
also raisethehypothesis that thereis consider- 
able redundancy in the encoding of motor 
parameters in each cortical area. In this con- 
text, one could conceive a potential coding 
schemein which the minimal neuronal mass 
required to gneratean appropriatearm/hand 
movement in agiven trial is selected froma 
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large pool of cortical neurons that modulate 
their firing before the onset of ahand move. 
ment (Principle 1). Asthis minimal neuronal 
mass could be defined by different combina- 
tions of individual neurons (Principle5, below) 
such acoding scheme would ensurethat reli- 
able motor outputs would continueto be pro- 
duced, even if significant numbers of neurons 
werelost owingto lesions of the motor system. 

The fourth principlestates that the physio- 
logical properties of cortical ensembles are 
adaptive and can changeas a function of expe- 
rienceand training, so the existence of cortical 
and subcortical plasticity must betaken into 
account in the design of an efficient BM |. 
Finally, thefifth principle asserts that thesame 
hand/arm movements can be produced by dis- 
tinct spatiotemporal patterns of neural ensem- 
blefiring. In other words, on asingle trial basis, 
different combinations of singleneurons from 
several cortical areas, producing distinct spa- 
tiotemporal sequences of neuronal firing, can 
encode the same movement. Experimental 
evidence supports three of these principles; the 
third and fifth principles arestill hypothetical. 
H owever, preliminary evidence obtained in 
our laboratory indicates that they have merit 
and should be investigated further. 

A BMI design that takes into account these 
five principles would use chronic implants of 
high-density microwire arraysto samplethe 
extracdlular activity of 100-200 neurons from 
each of 3-5 cortical areas simultaneously. 
Using this approach, several motor control 
parameters could besimultaneously extracted 
from therecorded neuronal sample, enabling 
patients to achieve more elaborate control of 
artificial actuators— such as robot arms and 
grippers — to recreate various arm/hand 
movements aimed at increasing their inde 
pendence (for example, feeding without assis- 
tance) or atimprovingtheir ability to interact 
with their surrounding environment (for 
example, controlling a whealchair). Additional 
improvements in the ability to control actua- 
torslocated in renoteenvironments (such as 
robots in different rooms) could further 
improve quality of life for these patients. 

Overall, thisBM I design would increase 
the chances of achieving robust, continuous 
performance and long-term reliability. By 
sampling from alarge neuronal population 
from the onset, the performance of such a 
BM 1 would bemuch less affected by eventual 
problems with individual electrodes, reduc- 
tions in neuronal samples or changes in 
the physiological properties of individual 
neurons. Indeed, loss of all but one of the 
implants could still provide enough informa- 
tion for the continuous operation of sucha 
cortical neuroprosthesis. 
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Conclusions 

During thelast five years, a series of experi- 
mental studies has demonstrated the feasibility 
of building neuroprosthetic devices to restore 
basic motor functions in patients suffering 
from catastrophic body paralysis. As obstacles 
to bringing these devices to the clinical arena 
areovercome, further research on BM Isisalso 
likely to sour the development of various new 
models to investigate the operation of the 
neural circuits that will be used asa source of 
brain signals to driveanew generation of neu- 
roprostheses. The confluence of thesetwo out- 
comes might lead to profound contributions 
to thestudy of distributed neural coding, and 
the design of new brain-controlled actuators 
aimed at minimizing the devastating motor 
impairments that are caused by alarge reper- 
toire of neurological disorders. 
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Since the original demonstration that electrical activity 
generated by ensembles of cortical neurons can be 
employed directly to control a robotic manipulator, 
research on brain-machine interfaces (BMIs) has experi- 
enced an impressive growth. Today BMls designed for 
both experimental and clinical studies can translate raw 
neuronal signals into motor commands that reproduce 
arm reaching and hand grasping movements in artificial 
actuators. Clearly, these developments hold promise for 
the restoration of limb mobility in paralyzed subjects. 
However, as we review here, before this goal can be 
reached several bottlenecks have to be passed. These 
include designing a fully implantable biocompatible 
recording device, further developing real-time computa- 
tional algorithms, introducing a method for providing 
the brain with sensory feedback from the actuators, and 
designing and building artificial prostheses that can be 
controlled directly by brain-derived signals. By reaching 
these milestones, future BMls will be able to drive and 
control revolutionary prostheses that feel and act like 
the human arm. 


Introduction 

Less than a decade ago, hardly anyone could have 
predicted that attempts to build direct functional 
interfaces between brains and artificial devices, such as 
computers and robotic limbs, would have succeeded so 
readily, and in the process would have led to the establish- 
ment of a new area at the frontier of systems neuroscience. 
Born as a highly multidisciplinary field, basic research on 
brain—machine interfaces (BMIs) has moved at a stunning 
pace since the first experimental demonstration in1999 
that ensembles of cortical neurons could directly control a 
robotic manipulator [1]. Since then, a continuous stream of 
research papers has kindled an enormous interest in BMIs 
among the scientific community and the lay public. This 
interest stems from the considerable potential of this 
technology for restoration of motor behaviors in severely 
handicapped patients. 

Indeed, BMIs have been primarily conceived as a poten- 
tial new therapy to restore motor control in severely dis- 
abled patients, particularly those suffering from 
devastating conditions such as amyotrophic lateral 
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sclerosis (ALS), spinal cord injury, stroke and cerebral 
palsy. As this technology advances and the risks of invasive 
brain recordings decrease, BMIs might also hold promise 
for amputees. In addition to the systems controlling upper- 
limb prostheses, BMIs dedicated to the restoration of 
locomotion and speech are likely to emerge. 

However, such stellar progress also breeds unrealistic 
expectations that such a future is just around the corner. 
Thus, the understandable eagerness in attaining the lofty 
goal of helping severely disabled patients has to be carefully 
calibrated by an objective analysis of the current state 
and future directions of the field. Such analysis indicates 
that, despite the optimism raised by a barrage of new 
accomplishments, there are still many issues that preclude 
straightforward translation of experimental BMIs into clin- 
ical applications. Indeed, most of the invasive BMIs have 
been tested only in experimental animals. Thus, despite 
recent enthusiasm to move emergent, and in some cases not 
thoroughly tested, BMI-related technology into clinical 
trials, much experimentation remains to be done before 
BMIs can become a safe and efficient rehabilitation tool. 

Here, we highlight some of the fundamental obstacles 
faced by BMI research and propose a series of milestones 
that can transform recent experimental advances into 
viable clinical applications in the next 10-20 years. The 
roadmap detailed here takes into account the recent his- 
tory of the field, the factors that influenced its growth, and 
a critical analysis of the published work. 


Non-invasive BMls 

Figure 1 depicts a classification of the BMIs (or 
brain—computer interfaces, BCIs) developed during the 
past decade. The first feature that distinguishes BMIs is 
whether they utilize invasive (i.e. intra-cranial) or 
non-invasive methods of electrophysiological recordings. 
Non-invasive systems primarily exploit electroencephalo- 
grams (EEGs) to control computer cursors or other devices. 
This approach has proved useful for helping paralyzed or 
‘locked in’ patients develop ways of communication with 
the external world [2-11]. However, despite having the 
great advantage of not exposing the patient to the risks of 
brain surgery, EEG-based techniques provide communica- 
tion channels of limited capacity. Their typical transfer 
rate is currently 5-25 bitss~* [2,11]. Although such a 
transfer rate might not be sufficient to control the move- 
ments of an arm or leg prosthesis that has multiple degrees 
of freedom, past and recent research in this field seems to 
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Brain—machine interfaces 


Y 


: 


Invasive 
Electrodes are implanted 


time, it carries risks associated 


Single recording site 


Several groups have built BMIs 
based on neural recordings from a 
single cortical area. A single-area 
BMI decodes neuronal activity 
specific for that area, for example 
motor commands in M1 or cognitive 
signals in PP. 


intracranially. This methodology 

provides neural signals of the best 
quality and has a high potential for 
further improvement. At the same 


with an invasive surgical procedure. 


Multiple recording sites 


By recording simultaneously from 
many areas, this approach takes 
advantage of distributed processing 
of information in the brain. Although 
technically challenging, it is highly 
promising for both developing BMIs 
and gaining fundamental knowledge. 


Non-invasive 


This methodology is based on the 
recordings of EEGs from the surface 
of the head. It provides solutions for 
paralyzed people for simple 
communications with the outside 
world. However, neural signals have 
a limited bandwidth. 


LFPs 
BMls based on decoding 


Small samples 


In certain cases, small groups of 
neurons are sufficient for providing 
control signals to a BMI. This design 
suffers from instability related to 
variability of neuronal activity and 
changes in the sampled populations 
of neurons. 


is, however, limited. 


LFPs suffer less from biocompatibility 
issues. Their advantage is that they 
reflect population effects such as 
neural oscillations. Their bandwidth 


Large ensembles 


Large neuronal ensembles 
(hundreds and in the future feasibly 
thousands of cells) provide a stable 
signal to control a multi-degree-of- 
freedom prosthesis. This approach 
instigates new computational 
solutions. 
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Figure 1. Classification of brain-machine interfaces. Abbreviations: BMI, brain machine interface; EEG, electroencephalogram; LFP, local field potential; M1, primary motor 


cortex; PP, posterior parietal cortex. 


indicate that EEG-based BMIs are likely to continue to 
offer some practical solutions (e.g. cursor control, commu- 
nication, computer operation and wheelchair control) for 
patients in the future. 

Original attempts to provide subjects with feedback 
signals derived from their own brain activity were made 
in the 1960s and 1970s. Primarily, these attempts were 
aimed at enabling human subjects to gain voluntary con- 
trol over brain rhythms. Nowlis and Kamiya claimed that, 
after training with an EEG biofeedback, human subjects 
acquired an ability to detect their own alpha rhythms [12], 
and this claim was later scrutinized by Plotkin [13]. 
Sterman and colleagues came to similar conclusions by 
utilizing the sensorimotor mu rhythm in cats [14] and 
humans [15]. In addition, Black operantly conditioned dogs 
to control their hippocampal theta rhythm [16]. Clearly, 
the results of these pioneering experiments helped pave 
the way for the introduction, a few years later, of EEG- 
based BCIs. 

Generally, EEG-based BCIs try to decipher the subject’s 
voluntary intentions and decisions through measurements 
of the combined electrical activity of massive neuronal 
populations. As such, both the spatial and temporal reso- 
lution of EEGs become limited owing to the overlapping 
electrical activity generated by different cortical areas. 
Furthermore, during the passive conductance of these 
signals through brain tissue, bone and skin, resolution is 
also lost owing to the low-pass filtering of the EEG signals. 
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EEGs are also susceptible to electromyographic (EMG), 
electrooculographic (EOG) and mechanical artifacts. 
Despite these well-known shortcomings, EEG techni- 
ques can detect modulations of brain activity that correlate 
with visual stimuli, gaze angle, voluntary intentions and 
cognitive states. These properties have led to development 
of several classes of EEG-based systems, which differ 
according to the cortical areas recorded, the features of 
EEG signals extracted, and the sensory modality providing 
feedback to subjects. One class of BCIs makes use of visual 
evoked potentials (VEPs). These BCIs detect the VEPs that 
occur when subjects look at particular items on a computer 
screen [17,18] or attend to them [19]. BCIs based on the 
P300 evoked potential uncover the subjects’ choices by 
distinguishing parietal cortex responses to the preferred 
versus non-preferred stimuli [20—22]. Several BCI designs 
continuously drive computer cursors. Both slow cortical 
potentials, recorded over several cortical areas [23], and 
faster mu (8-12Hz) and beta (18-26Hz) rhythms, 
recorded over sensorimotor cortex [24-26], have been 
exploited in such BCIs. For example, one such system 
relies on event-related synchronization and desynchroni- 
zation of the EEGs associated with motor imagery [25,27]. 
Training to operate EEG-based BCIs can take many 
days [2]. Visual feedback is the essential part of such 
training. Some BCI designs rely on the subjects’ ability 
to develop control of their own brain activity using biofeed- 
back, whereas others utilize classifier algorithms that 
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recognize EEG patterns related to particular voluntary 
intentions. Recently, adaptive algorithms that constantly 
update the classifier parameters during training have been 
implemented [26]. 

Several strategies have also been proposed to provide 
feedback to users of EEG-based BCIs. For instance, 
virtual-reality systems can provide a realistic feedback 
that can be efficient for BCI training [28]. In a recent 
demonstration of this approach, subjects navigated 
through a virtual environment by imagining themselves 
walking [29]. 

In an effort to improve the resolution of brain potentials 
monitored by the BCIs, more invasive recording methods, 
such as electrocorticograms (ECoGs) recorded by subdural 
electrodes, have been introduced. ECoGs sample neuronal 
activity from smaller cortical areas than conventional 
EEGs. In addition, they contain higher-frequency gamma 
rhythms (>30 Hz). Consequently, ECoG-based BCIs are 
expected to have better accuracy and shorter training 
times than BCIs based on EEGs [30]. 

EEG-based BCIs have been implemented as solutions 
for patients suffering from various degrees of body paraly- 
sis. These BCIs (in the case of patients with advanced ALS) 
enable control of computer cursors, which the patients use 
to communicate with the external world or to indicate their 
intentions. The first successful and most well received 
application of such an approach was based on the utiliza- 
tion of slow cortical potentials to control a computer-aided 
spelling system [3,31]. 

BClIs based on mu and beta rhythms have also been 
tested in severely paralyzed people [32]. One study 
reported that a tetraplegic patient, aided by a BCI that 
detected beta waves in his sensorimotor cortex and 
activated a functional electrical stimulation device, 
learned to grasp objects using his paralyzed hand [83]. 
A motor imagery-based system [32], coupled to an 
implanted neuroprosthesis system [34] (Freehand®) 
has been used to help a partially paralyzed patient. In 
addition, tetraplegic patients were able to gain some 
degree of control of the P300-based BCI [21]. Off-line 
analyses showed that P300 potentials can be used to 
obtain information about stimulus selections made by 
patients with ALS [22]. 

In addition to using EEGs, imaging techniques such as 
functional magnetic resonance imaging (fMRI), have been 
explored as a new source of brain-derived signals to drive 
BCIs [35]. Although fMRI-based BCIs are not suitable for 
everyday use and suffer from temporal delays of several 
seconds, they have good spatial resolution and, most 
importantly, can sample the activity of deep brain struc- 
tures. Recently, {MRI was used to measure brain activation 
during the operation of a BCI based on slow cortical 
potentials [4]. 

Myolectric systems that make use of voluntary activa- 
tions of unaffected muscles in partially paralyzed subjects 
and amputees [36-39], and use these signals to control 
limb prostheses and exoskeletons, present an alternative 
to the existing non-invasive BCIs. Currently, these sys- 
tems are more practical for everyday situations than EEG- 
based BCIs [11]. The details of their operation are beyond 
the scope of this review. 
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In summary, severely and partially paralyzed patients 
can reacquire basic forms of communication and motor 
control using EEG-based systems. Yet motor recovery 
obtained using these systems has been limited, and no 
clear breakthrough that could significantly enhance the 
power of EEG-based BCIs in the near future has been 
reported in the literature [11]. This by no means reduces 
the clinical utility of such systems. Some of them have 
improved the quality of life of patients, such as the BCI for 
spelling [3]. But if the goal of a BMI is to restore move- 
ments with multiple degrees of freedom through the con- 
trol of an artificial prosthesis, the message from published 
evidence is clear: this task will require recording of high- 
resolution signals from the brain, and this can be done 
using invasive approaches. 


Invasive BMIs 

Invasive BMI approaches are based on recordings from 
ensembles of single brain cells (also known as single units) 
or on the activity of multiple neurons (also known as 
multi-units). These approaches have their roots in the 
pioneering studies conducted by Fetz and colleagues in 
the 1960s and 1970s [40-45]. In these experiments, 
monkeys learned to control the activity of their cortical 
neurons voluntarily, aided by biofeedback indicating the 
firing rate of single neurons. A few years after these 
experiments, Edward Schmidt raised the possibility that 
voluntary motor commands could be extracted from raw 
cortical neural activity and used to control a prosthetic 
device designed to restore motor functions in severely 
paralyzed patients [46]. 

Largely owing to technical difficulties associated with 
obtaining the needed cortical signals and implementing 
real-time interfaces quickly enough, thorough experimen- 
tal testing of Schmidt’s proposition took almost two dec- 
ades to be accomplished. These bottlenecks were passed 
because of a series of experimental and technological 
breakthroughs that led to a new electrophysiological meth- 
odology for chronic, multi-site, multi-electrode recordings 
[47-51]. The BMI approach that relies on long-term record- 
ings from large populations of neurons (100-400 units) 
evolved from experiments carried out in 1995 [47]. After 
the introduction of such an approach, a series of studies 
demonstrated that neuronal readout of tactile stimuli 
could be uncovered using pattern-recognition algorithms, 
such as artificial neural networks [52,53]. These develop- 
ments paved the way for the first experiment in which 
neuronal population activity recorded in behaving rats 
enacted movements of a robotic device that had a single 
degree of freedom [1]. Soon after this first demonstration, a 
similar BMI approach was shown to work in New World 
[54] and rhesus monkeys [55-58]. As a result of these 
experimental efforts, in less than six years several labora- 
tories reported BMIs that reproduced primate arm reach- 
ing [1,54-58] and the combination of reaching and grasping 
movements [57], using either computer cursors or robotic 
manipulators as actuators. 

During the past three years, most of the published 
studies on BMIs have been conducted in behaving rhesus 
monkeys. There are several important differences 
that distinguish these BMIs (Figure 1). These include: 
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the number of cortical implants (e.g. uni-site or multi-site 
recordings); the cortical location of implants (e.g. frontal or 
parietal cortex, or both); the type of neural signal recorded 
(local field potentials versus single-unit or multi-unit sig- 
nals); and the size of the neural sample. With the exception 
of the BMIs used at Duke University (http://www. 
duke.edu/), all BMIs tested in monkeys have relied on 
single cortical site recordings either of local field potentials 
[59-62] or of small samples (<30) of neurons or multi-units 
[55,56,63]. Most of these small-sample, single-area BMIs 
utilized neural signals recorded in the primary motor 
cortex [55,56], although one group has focused on BMIs 
that processed neural signals recorded in the posterior 
parietal cortex [64]. At Duke University, a BMI strategy 
has recently been implemented based on single-unit 
recordings made during intra-operative placement of 
deep-brain stimulators in Parkinsonian patients [65]. 


Principles of BMI operation 

Invasive BMIs rely on the physiological properties of indi- 
vidual cortical and subcortical neurons (or pools of neu- 
rons) that modulate their activity in association with 
movements. First documented four decades ago by Evarts 
[66-68], such modulations are highly variable, from neu- 
ron to neuron and from trial to trial [69-72]. Thus, as much 
as neighboring neurons might display highly distinct firing 
modulation patterns during the execution of a particular 
movement, single-neuron firing can vary substantially 
from one trial to the next, despite the fact that the overt 
movements remain virtually identical. Yet averaging 
across many trials reveals fairly consistent firing patterns. 
By the same token, averaging across large populations of 
neurons significantly reduces the variability of signals 
derived from single neurons [54,69]. 

Extracting motor control signals from the firing patterns 
of populations of neurons and using these control signals 
to reproduce motor behaviors in artificial actuators are the 
two key operations that a clinically viable BMI should 
perform flawlessly [51,73]. To be accepted by patients, 
BMI devices will also have to act in the same way and feel 
the same as the subjects’ own limbs. Recent findings suggest 
that this task might be accomplished by creating conditions 
under which the brain undergoes experience-dependent 
plasticity and assimilates the prosthetic limb as if it were 
part of the subject’s own body. Until recently, such plasticity 
was achieved using visual feedback. However, a more 
efficient way to assimilate the prosthetic limb in the brain 
representation could be to use multiple artificial feedback 
signals, derived from pressure and position sensors placed 
on the prosthetic limb. These feedback signals would 
effectively train the brain to incorporate the properties 
of the artificial limb into the tuning characteristic of 
neurons located in cortical and subcortical areas that 
maintain representations of the subject’s body. We predict 
that such plasticity will result in sensory and motor areas 
of the brain representing the prosthetic device. 


A proposed roadmap for the future of BMI research 

To achieve the ambitious goal of creating a clinically useful 
invasive BMI for restoring upper-limb mobility, one has to 
pass the following key bottlenecks: 
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e Obtaining stable, very long-term recordings (i.e. over 
years) of large populations of neurons (i.e. hundreds 
to thousands) from multiple brain areas. This task 
encourages development of a new generation of 
biocompatible 3D electrode matrices that yield 
thousands of channels of recordings while producing 
little tissue damage at implantation and minimal 
inflammatory reaction thereafter. 

e Developing computationally efficient algorithms, 
that can be incorporated into the BMI software, for 
translating neuronal activity into high-precision 
command signals capable of controlling an artificial 
actuator that has multiple degrees of freedom. 

e Learning how to use brain plasticity to incorporate 
prosthetic devices into the body representation. This 
will make the prosthetic feel like the subject’s own 
limb. 

e Implementing a new generation of upper-limb 
prosthetics, capable of accepting brain-derived 
control signals to perform movements with multiple 
degrees of freedom. 

We now discuss some potential avenues for addressing 
the first three of these major challenges. A thorough 
discussion of the fourth challenge (i.e. engineering a new 
generation of prosthetic arms) is beyond the scope of this 
review. 


Long-term recordings of neuronal activity from multiple 
brain areas 

Although recording from single neurons is the first choice 
of neurophysiologists, multi-unit signals that comprise 
activity of a few neurons can also be efficiently used in 
BMI control [57]. In addition, several reports have sug- 
gested using local field potentials [59-62]. It is conceivable 
that, in future, advanced neuroprosthetic devices will use 
hybrid solutions in which a combination of several types of 
neural signals are recorded and processed. Here, however, 
we focus on using single-unit and multi-unit signals as the 
primary input to a BMI. This choice raises a fundamental 
question: how many neurons does a BMI need to sample to 
produce effective motor outputs? 

This question, first raised several years ago [51,73], 
remains a matter of debate. Some groups [55,56,63] have 
strongly claimed that recordings from a small number of 
neurons can be sufficient for good performance of a BMI. 
Selected populations of highly tuned neurons can indeed 
accurately predict movement parameters [74]. However, 
highly tuned neurons are rare in a typical random sample 
of cortical cells. Given that the neuronal yield of all chronic 
recording techniques is produced by random sampling of 
neurons, it is unrealistic to expect that a large fraction of 
these cells will be highly tuned to a particular motor 
variable. Moreover, it would be even more unrealistic to 
expect that a small neuronal sample would represent 
several variables of interest. Therefore, large samples of 
recorded neurons are preferable, at the very least to enable 
selection of a sufficient number of highly tuned neurons. 
Besides, the reason for relying on large neuronal popula- 
tions goes far beyond the issue of selecting the best per- 
forming cells. Both the accuracy [54,57,70] and the 
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reliability [69] of predictions improve considerably with 
the number of simultaneously recorded neurons, because 
motor information seems to be represented in the cortex in 
a highly distributed way. Thus, as the neuronal sample 
increases in size, errors related to individual neuron firing 
variability decrease significantly [69]. So looking into the 
future, it seems unlikely that invasive BMIs based on a 
small group of neurons will be capable of continuously 
reproducing in artificial limbs the range of fine movements 
normally performed by the human arm and hand. 

Currently, chronically implanted microwire arrays offer 
the best compromise between safety, recording longevity 
and neuronal yield required to operate BMIs [48,51, 
54,57,58,73]. It is clear that this methodology will continue 
to be applied in experimental settings, but several signifi- 
cant improvements are required before it becomes fully 
applicable for long-term (months to years) chronic clinical 
applications in humans. First and foremost, the broad and 
challenging issue of biological compatibility [75-80] has 
to be properly addressed and solved. Second, fully 
implantable technologies using wireless headstages for 
amplification of neuronal signals have to be implemented 
to reduce the risks of infection introduced by the use of 
cables that connect brain implants to external hardware. 

Current microelectrode designs typically enable good 
quality recordings to be made for several months. In 
certain cases and species, these recordings can last for 
several years [81]. However, recording quality often dete- 
riorates, probably owing to a process of electrode encapsu- 
lation by fibrous tissue and cell death in the vicinity of the 
electrode [77]. Some authors have proposed that electrodes 
that contain neurotrophic medium [82-85], or are coated 
with factors that promote neuronal growth (e.g. nerve 
growth factor, brain-derived neurotrophic factor or lami- 
nin) and various anti-inflammatory compounds (e.g. dex- 
amethazone) [77,86-91], might be a way to cope with 
encapsulation. Currently, it is unclear whether these 
approaches will be useful. 

Efforts to resolve the biocompatibility issues will prob- 
ably have to be pursued in parallel with the development of 
new 3D electrode matrices, which should aim to increase 
the average yield to thousands of neuronal signals per 
implanted probe. Current alternatives to such microwires 
(e.g. the Utah probe [92], which implements arrays of rigid, 
single-ended electrodes) have yet to prove their adequacy 
to support the haste in which this technology was moved 
into clinical applications. Judging from the published evi- 
dence, such arrays are best suited to sample neuronal 
activity from flat surfaces of cortical gyri in animal experi- 
ments. However, this design might not be suitable for long- 
term use in human patients. In addition to issues of how 
electrodes are inserted into the cortex, the inability to 
sample from deep cortical layers, and many unanswered 
biocompatibility questions, the reliability of the recording 
system utilized by currently available probes is also com- 
promised by the continuous stress of a daily routine that 
involves external cables and plugging and unplugging of 
external head-stages. These operations carry a risk of 
causing tissue damage, bleeding and brain infection. 
Such a risk of failure, which might be tolerable in animal 
experiments, is unwanted in practical applications 
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for humans. Cyberkinetics Neurotechnology Systems 
(http://www.cyberkineticsinc.com/content/index.jsp) has 
recently started clinical trials in severely paralyzed 
patients of a BCI based on a probe developed at the 
University of Utah (http:/Wwww.utah.edu/). Because no 
peer-reviewed publication has appeared related to this 
work, the exact outcome of this study remains unknown. 

From these considerations, it is clear that the issues 
related to the long-term functionality of implantable micro- 
electrodes, and to the development of fully implantable 
electronic devices for amplification of a large number of 
neuronal signals and their wireless transmission to the 
actuator, are the major technological challenges that will 
determine the success or failure of future clinical applica- 
tions of BMI technology. These technological developments 
are necessary not only to increase the practical usefulness 
of the BMI (more neurons mean better stability and accu- 
racy) but also to ensure that risks to patient health are 
minimized. Auspiciously, telemetry transmission methods 
[93-97] for effective wireless transmission of multi- 
channel neuronal signals have already started to appear 
in the literature [98,99]. These solutions are currently 
being tested in animal experiments. 

Many new ideas of how to improve neuronal recordings 
have been proposed recently. These range from ceramic- 
based multi-electrode arrays [100] to nanotechnology 
probes that access the brain through the vascular system 
[101]. In this latter design, probes record neuronal activity 
without compromising brain parenchyma. Undoubtedly, 
much more testing will be needed to conclude which of 
these ideas are viable. 


Developing algorithms for translating neuronal activity 
into command signals for artificial actuators 

Currently, neuroscientists are far from obtaining a clear 
understanding of how motor and cognitive information is 
processed by the populations of neurons that form large 
brain circuits. Rate encoding, temporal encoding and popu- 
lation encoding principles have been suggested, and var- 
ious experimental paradigms, including BMIs, have been 
developed to test the validity of these concepts. However, 
precise knowledge of computations performed by brain 
circuits is not crucial for the construction of clinically 
relevant BMIs. Mostly, BMI platforms take advantage of 
the well known correlation between discharges of cortical 
neurons and motor parameters of interest, and perform a 
reverse operation: they predict motor parameters from 
patterns of neuronal firing. Generally, predictions of motor 
parameters do not signify a causal relationship between 
the neuronal activity and the generation of movements. 
One type of correlation between neuronal activity and 
movement is known as directional tuning [102,103], and 
correlations of neuronal activity with kinematic [104—106] 
and kinetic [107-109] parameters of movements have also 
been described. 

Although a wealth of linear and nonlinear algorithms 
for translating neuronal activity into commands to artifi- 
cial actuators have been suggested [1,54,56,57,70,110— 
116], relatively simple multiple linear regression models 
have proved to be efficient in many practical BMI designs 
[54,55,57,58,65,117]. In these models, predicted motor 
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parameters are derived from the weighted sums of 
neuronal rates of firing, measured at different time points 
in the past. The number of regressors in the model and the 
time window used for predictions can be optimized for each 
concrete BMI application [70,74,117]. Linear methods, 
such as population vector predictions, can incorporate 
adaptive algorithms that continuously update the model 
parameters while the subject trains [56]. 

Basic research using the BMI paradigm has lent sup- 
port to some fundamental principles of neural information 
coding. For example, studies in which several independent 
linear models were run in parallel revealed that several 
motor parameters, such as arm position, velocity, accel- 
eration and hand gripping force, could be predicted simul- 
taneously by separate combinations of the activity of the 
same original pool of neurons [57]. This finding supports 
the notion that multiple motor parameters are processed 
by overlapping neuronal ensembles. As a corollary, a single 
cortical neuron can contribute to several predictions 
simultaneously. 

The choice of motor parameters extracted in future 
clinical BMIs will depend on the main rehabilitation or 
therapeutic goals of these applications. For example, an 
experimental BMI for reaching and grasping [57] predicted 
hand velocity and gripping force that matched the char- 
acteristics of a robotic arm equipped with a gripper. In the 
near future, this design could lead to the implementation of 
neuroprosthetic devices that help quadriplegic or ‘locked 
in’ patients to reach and grasp objects in the surrounding 
space. We can also envisage that BMIs that synthesize 
speech, based on neuronal signals recorded in intact 
speech-related regions, could one day help patients suffer- 
ing from aphasia due to cortical strokes recover their 
ability to communicate. 

Future clinical applications might also take advantage 
of BMIs that predict EMG signals [117]. The main benefit 
of this design compared with a BMI that predicts kinematic 
parameters is that the signals of individual muscles can 
control biologically-inspired devices, which would produce 
a whole range of actuator stiffness. This is an important 
property needed for a future generation of prosthetic limbs 
that should be able to manipulate objects with different 
physical characteristics. Another powerful future applica- 
tion for BMIs that decode EMGs is the construction of 
brain—muscle interfaces that directly stimulate the mus- 
cles of paralyzed patients and thereby restore mobility by 
using the patient’s own musculoskeletal apparatus 
[37,118,119]. Such BMIs are likely to be much more accep- 
table to patients, particularly because the hardware 
needed for amplification, transmission and processing of 
brain-derived control signals, and the muscle stimulators 
driven by these neural signals, such as the BION [120], can 
be entirely encased in the patient’s body. In the future, it is 
conceivable that such BMIs could merge the current corti- 
cally driven paradigm with methods and new technologies 
developed in the field of functional electrical stimulation. 

Early BMI designs focused on decoding motor para- 
meters from neuronal activity [1,54-56,117]. More 
recently, it was suggested that BMIs that decode cognitive 
signals, for example those that decode intended reach 
direction during the delay periods preceding movement 
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execution [64,121—123], could also be efficient. Although 
this idea is very attractive, a BMI based exclusively on 
cognitive signals cannot execute continuous control of 
movement parameters. Instead, it decodes higher-order 
characteristics of movements, such as reach direction or 
characteristics of objects being grasped, and delegates 
lower-order details of motor execution to the actuator 
controller. Recently, we have proposed that a hybrid 
BMI, based on a shared control mode of operation [124], 
can improve the accuracy with which the prosthetic imple- 
ments the voluntary intentions of the subject. A shared- 
control mode of operation would be achieved by a combina- 
tion of high-order brain-derived signals, conveying the 
subject’s voluntary intentions, and low-level artificial 
‘reflex-like’ circuits, designed to improve the precision with 
which prosthetic limb movements are generated. 

In the future, BMIs that take advantage of the higher- 
order neuronal representations of movement-related 
variables will also emerge. These BMIs will probably 
derive information from representations of movement 
sequences [125,126], reference frames [127-130], 
potential movement targets [131] and simultaneous 
encoding of multiple spatial variables, such as movement 
direction, orientation of selective spatial attention, and 
gaze angle [132,133]. Utilization of this wide array of 
information will endow BMIs with a much more flexible 
control of prosthetic limbs. 

In the same context, we also believe that future BMIs 
will take advantage of new insights on how neural circuits 
encode temporal characteristics of movements. Recent 
studies [134-137] have indicated that a rather distributed 
representation of temporal information might exist in the 
brain. Recordings obtained from primary motor and pre- 
motor cortical ensembles while monkeys performed self- 
timed button presses [138] enabled prediction of both the 
time that had elapsed since the monkey pressed the button 
and the time until the button would be released. Because 
any motor behavior has a temporal structure, and because 
episodes of movement execution are typically intermingled 
with periods of immobility during which movements are 
being prepared, a BMI that decodes behavioral time will be 
able to inhibit movements of the actuator during waiting 
periods and release the actuator at appropriate times. 
These operations will be based on the voluntary intentions 
of the user. 


Making use of brain plasticity to incorporate prosthetic 
devices into the body representation 

Controlling an artificial actuator through a BMI can be 
thought of as a process somewhat similar to the operation 
required by subjects to operate tools — a capacity that is 
inherent only in higher primates such as chimpanzees and 
humans [139]. Almost 100 years ago [140], Head and 
Holmes suggested that the ‘body schema’— that is, the 
internal brain representations of one’s body — could extend 
itself to include a wielded tool. This idea was validated by 
the experimental demonstration that cortical neurons 
extend their visual receptive fields along the length of a 
rake used by monkeys to retrieve distant objects [141]. 
Psychophysics experiments also support the notion that 
tool usage leads to remapping of the ‘body schema’ in 
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humans [142,143]. Accordingly, a recent neuroimaging 
study [144] described specific activations of the right 
ventral premotor cortex during manipulation of a myo- 
electric prosthetic hand. Altogether, these results suggest 
that long-term usage of an artificial actuator directly 
controlled by brain activity might lead to substantial 
cortical and subcortical remapping. As such, this process 
might elicit the vivid perceptual experience that the arti- 
ficial actuator becomes an extension of the subject’s body 
rather than a mere tool. This suggestion is supported by 
the report of primary sensorimotor cortex activation dur- 
ing perceived voluntary movements of phantom limbs in 
amputees [145]. 

Perhaps the most stunning demonstration of tool assim- 
ilation by animals was observed when both rats and pri- 
mates learned to operate an actuator through a BMI, 
without the need to move their own limbs [1,56—58]. In 
these experiments, decoding algorithms were initially 
trained to predict limb movements of animals from the 
activity of neuronal populations. Remarkably, after these 
animals started to control the actuator directly using their 
neuronal activity, their limbs eventually stopped moving, 
while the animals continued to control the actuator by 
generating proper modulations of their cortical neurons. 
Interestingly, during these episodes neuronal tuning to 
movements of the subject’s own limbs decreased while the 
animals continued to control the artificial actuator by their 
brain activity [58]. The most parsimonious interpretation 
of this finding is that the brain was capable of undergoing a 
gradual assimilation of the actuator within the same maps 
that represented the body [57,58]. Neuronal mechanisms 
mediating such plasticity are far from being understood. 
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However, it is fair to state that there is a growing 
consensus in the literature that continuous BMI 
operations in primates lead to physiological changes in 
neuronal tuning, which include changes in preferred 
direction and direction tuning strength of neurons 
[56-58]. In addition, broad changes in pair-wise neuronal 
correlation can be detected after BMIs are switched to 
operate fully under brain-control mode [57,58]. 

Along with these physiological adaptations of 
neuronal firing patterns, behavioral performance 
improves as animals learn to operate BMIs effectively 
[56-58]. Initial training to operate a BMI is character- 
ized by an increase in neuronal firing rate variance, 
which cannot be simply explained by changes in limb 
or actuator movements [146]. As the quality of BMI 
control improves, initial elevation of neuronal firing 
variability subsides. Plastic changes in neuronal firing 
patterns during BMI control, leading to the physiological 
incorporation of the artificial actuator properties into 
neuronal space, could account for these changes in firing 
rate variance. This interpretation is in accord with the 
theory of optimal feedback control [147-149]. According 
to this theory, a motor system acts as a stochastic feed- 
back controller that optimizes only those motor para- 
meters that are necessary to achieve the goals of a 
particular task. During the brain-control mode of opera- 
tion of a BMI, the goals of a motor task are achieved only 
by direct brain control of an artificial actuator. Thus, in 
terms of optimal feedback control theory, neuronal 
ensembles should adapt their physiological tuning prop- 
erties to represent better the goal-related variables of 
the task performed by the BMI. 
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Figure 2. A BMI with multiple feedback loops being developed at the Duke University Center for Neuroengineering. A rhesus macaque is operating an artificial robotic 
manipulator that reaches and grasps different objects. The manipulator is equipped with touch, proximity and position sensors. Signals from the sensors are delivered to 
the control computer (right), which processes them and converts to microstimulation pulses delivered to the sensory areas in the brain of the monkey, to provide it with 
feedback information (red loop). A series of microstimulation pulses is illustrated in the inset on the left. Neuronal activity is recorded in multiple brain areas and translated 
to commands to the actuator, via the control computer and multiple decoding algorithms (blue loop). Arm position is monitored using an optical tracking system that tracks 
the position of several markers mounted on the arm (green loop). We hypothesize that continuous operation of this interface would lead to incorporation of the external 
actuator into the representation of the body in the brain. Figure designed by Nathan Fitzsimmons. 
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Figure 3. How a fully-implantable BMI could restore limb mobility in paralyzed 
subjects or amputees. Although the details of this system have to be worked out 
through future research, it is clear that the BMI for human clinical applications 
should be encased in the patient’s body as much as possible. Wireless telemetry 
offers a viable solution for this purpose. The prosthesis not only should have the 
functionality of the human arm in terms of power and accuracy of the actuators, 
but also should be equipped with the sensors of touch and position from which 
signals can be transmitted back to the subject’s brain. 


Making the prosthetic feel like the subject’s own limb 
using microstimulation of cortical sensory areas 
Peripheral tactile and proprioceptive signals contribute to 
the normal operation of one’s limbs and the perception that 
they are part of the body [142,143]. For a neuroprosthesis 
to behave and feel as a natural appendage of the subject’s 
body, it will have to be instrumented with various sensors 
that can provide multiple channels of ‘sensory’ information 
back to the subject’s brain. In most current BMI designs, 
animal subjects receive sensory information from the 
actuator through visual feedback [55-58]. Predictions of 
motor parameters are less stable in the absence of visual 
feedback [70] than when it is present [55-58]. 

Curiously, the use of tactile and proprioceptive-like 
feedback in BMI research remains largely unexplored. 
Recently, in collaboration with John Chapin, we have 
started to explore the intriguing possibility of delivering 
such sensory feedback information, generated in the actua- 
tor, to the brain through the use of multi-channel micro- 
stimulation of somatosensory cortical areas (Figure 2). 
Previous studies have shown that monkeys sense micro- 
stimulation patterns and can use them to guide their 
behavioral responses [150,151]. In a recent long-term 
study, owl monkeys could learn to guide their reaching 
movements by decoding vibratory stimuli applied to their 
arms [152]. Next, instead of vibratory stimulation, match- 
ing patterns of microstimulation were applied through the 
electrodes implanted in the primary somatosensory cortex 
[153]. Monkeys were still able to interpret correctly the 
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instructions provided by cortical microstimulation, and 
their behavioral performance eventually surpassed the 
level of performance observed when the vibratory stimulus 
was applied to their skin. These results suggest that 
cortical microstimulation might become a useful way to 
deliver long-term feedback from prosthetic limbs con- 
trolled by a BMI, and might contribute to the development 
of a completely new generation of neuroprosthetic devices 
for restoring various motor behaviors in severely impaired 
patients. 


Concluding remarks 

Our vision of neuroprosthetic developments that might 
emerge in the next 10-20 years includes a fully implan- 
table recording system that wirelessly transmits multiple 
streams of electrical signals, derived from thousands of 
neurons, to a BMI capable of decoding spatial and temporal 
characteristics of movements and intermittent periods of 
immobility, in addition to cognitive characteristics of the 
intended actions (Figure 3). This BMI would utilize a 
combination of high-order motor commands, derived from 
cortical and subcortical neuronal activity, and peripheral 
low-level control signals, derived from artificial ‘reflex-like’ 
control loops. Such a shared-control mode of BMI operation 
would either command an actuator with multiple degrees 
of freedom or directly stimulate multiple peripheral nerves 
and muscles through implantable stimulators. Highly 
instrumented artificial actuators, containing arrays of 
touch and position sensors, would generate multiple 
streams of sensory feedback signals that could be directly 
delivered to cortical and/or subcortical somatosensory 
areas of the subject’s brain, through spatiotemporal pat- 
terns of multi-channel microstimulation. Such closed-loop, 
hybrid BMIs would get one step closer to the dream of 
restoring a large repertoire of motor functions to a multi- 
tude of patients who currently have very few options for 
regaining their mobility. 
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OPINION 


Principles of neural ensemble 
physiology underlying the operation 
of brain—machine interfaces 


Miguel A. L. Nicolelisand Mikhail A. Lebedev 


Abstract | Research on brain—machine interfaces has been ongoing for at least a 
decade. During this period, simultaneous recordings of the extracellular electrical 
activity of hundreds of individual neurons have been used for direct, real-time 
control of various artificial devices. Brain—machine interfaces have also added 
greatly to our knowledge of the fundamental physiological principles governing 
the operation of large neural ensembles. Further understanding of these principles 
is likely to have a key role in the future development of neuroprosthetics for 
restoring mobility in severely paralysed patients. 


Recent demonstrations of direct, real-time 
interfaces between living brain tissue and 
artificial devices, such as computer cursors, 
robots and mechanical prostheses, have 
opened new avenues for experimental and 
clinical investigation’. Interest in these 
brain—machine interfaces (BMIs) has been 
kindled by the contribution that they may 
make to the treatment or rehabilitation of 
patients suffering from severe motor dis- 
abilities**?'*"'”. As such, BMIs have rapidly 
become incorporated into the development 
of ‘neuroprosthetics, devices that use neuro- 
physiological signals from undamaged com- 
ponents of the central or peripheral nervous 
system to allow patients to regain motor 
capabilities. Indeed, several findings already 
point to a bright future for neuroprosthet- 
ics in many domains of rehabilitation 
medicine®!***. For example, scalp electro- 
encephalography (EEG) signals linked to a 
computer have provided ‘locked-in’ patients 
with a channel of communication®'?. 
BMI technology, based on multi-electrode 
single-unit recordings — a technique 
originally introduced in rodents**** and 
later demonstrated in non-human pri- 
mates!?"!"1??7-5 — has yet to be transferred 
to clinical neuroprosthetics. Human trials 
in which paralysed patients were chroni- 
cally implanted with cone electrodes? or 


intracortical multi-electrode arrays** allowed 
the direct control of computer cursors. 
However, these trials also raised a number 
of issues that need to be addressed before 
the true clinical worth of invasive BMIs can 
be realized®. These include the reliability, 
safety and biocompatibility of chronic brain 
implants and the longevity of chronic 
recordings, areas that require greater atten- 
tion if BMIs are to be safely moved into the 
clinical arena***. 

BMIs provide new insights’**’ into 
important questions pertaining to the 
central issue of information processing by 
the CNS during the generation of motor 
behaviours*. Many recent review articles 
have covered BMI methods®'®>*? and 


...In addition to offering hope 
for a potential future therapy 
for the rehabilitation of severely 
paralysed patients, BMIs can 

be extremely useful platforms 
to test various ideas for how 
populations of neurons encode 
information in behaving animals. 


their potential implementation in medical 
rehabilitation'* 7°? 75, and so these issues 
will not be covered here. Instead, we focus 
on how modern BMI research has led to the 
proposal, and in some cases validation, of 
various physiological principles governing 
the operation of large populations of corti- 
cal neurons in behaving mammals (animals 
performing a given action or movement). 


Neuronal ensemble recordings 

Although the first multi-electrode recording 
experiments in rhesus monkeys date back to 
the mid 1950s, the current neurophysio- 
logical approach for sampling the extracellu- 
lar activity of large populations of individual 
neurons in behaving animals emerged in the 
early 1980s°-”. At that time, most of the sys- 
tems neuroscience community considered 
the single neuron to be the key functional 
unit of the CNS and, therefore, the main tar- 
get for neurophysiological investigation’. 
Not surprisingly, the transition to neural 
ensemble recordings was slow and difficult. 
In addition to the enormous technological 
and technical barriers, few systems neuro- 
physiologists saw any advantage in invest- 
ing effort and resources into this paradigm 
shift. As a result, the concept of population 
coding”*”,, first proposed by Young” and 
further popularized by Hebb’, played a 
distant second fiddle to the single-neuron 
doctrine’’”* for many decades. 

Today, the weight of evidence supports 
the idea that distributed ensembles of neu- 
rons define the true physiological unit of 
the mammalian CNS”*****, However, this 
does not mean that neurophysiologists have 
given up examining the degree to which 
animal behaviour can be affected by single- 
neuron activity®”°. Significant examples of 
the importance of single-neuron physiology 
to BMI research include the demonstration 
that single neurons can be conditioned to 
produce particular firing patterns if their 
activity is presented to primates as sensory 
feedback*!""’. In these experiments, the firing 
of single cells became so well correlated to 
the desired motor output that primates could 
use this single-neuron activity to control the 
movements of a gauge needle” or drive a 
functional electrical stimulator to produce 
an isometric contraction”. 
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During the past 25 years, the intro- 
duction of various new electrophysio- 
logical? 3848 .-780-99 and imaging 
methods’ has allowed neurophysiol- 
ogists to measure the concurrent activity 
of progressively larger samples of single 


neurons in behaving animals. Interestingly, 


the emergence of multi-electrode record- 


ings as a new electrophysiological paradigm 


Tactile and 
proprioceptive 
feedback 


Visual 
feedback 


occurred in parallel with the development 
of BMIs. As researchers started to implant 
more than one micro-electrode in the brain, 
it was proposed that single-neuron record- 
ings from the motor cortex might one day 
provide the source of signals to drive artifi- 
cial devices designed to restore mobility in 
paralysed patients''*. However, almost two 


decades went by before the first experiments 


Real-time 
analysis of 
brain activity 
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telemetry 
interface 
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Figure 1 | Principles of a brain-machine interface. a| A schematic of a brain-machine interface 
(BMI) for reaching and grasping. Motor commands are extracted from cortical sensorimotor areas 
using multi-electrode implants that record neuronal discharges in large ensembles of cortical cells. 
Signal-processing algorithms convert neuronal spikes into the commands to a robotic manipulator. 
Wireless telemetry can be used to link the BMI to the manipulator. The subject receives visual and 
somatosensory feedback from the actuator, possibly through the microstimulation of cortical sensory 
areas. b| Neuronal dropping curves for the prediction of arm movements in rhesus macaques’ calcu- 
lated for the ensembles recorded in different cortical areas: the dorsal premotor cortex (PMd), the 
primary motor cortex (M1), the primary somatosensory cortex ($1), the supplementary motor area 
(SMA) and the posterior parietal cortex (PP). Neuronal dropping curves describe the accuracy (R’) of 
a BMI’s performance as a function of the size of the neuronal ensemble used to generate predictions. 
The best predictions were generated by the M1. Prediction accuracy improved with the increase of 
neuronal ensemble size. ¢ | Predictions of hand gripping force calculated from the activity of the same 
cortical areas as in part a. Image in part a is modified, with permission, from REF. 8 © (2001) Macmillan 
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were conducted to test the hypothesis* that 
highly distributed populations of broadly 
tuned neurons can sustain the continuous 
production of motor behaviours in 
real-time!!!) 

FIGURE 1a shows a basic BMI paradigm® 
in which the kinematic and dynamic param- 
eters of upper- or lower-limb movements are 
predicted (or extracted) in real time from 
neuronal ensemble activity recorded by 
micro-electrode brain implants. In this con- 
text, the term prediction refers to the use of 
combined electrical neural ensemble activ- 
ity to estimate time-varying kinematic and 
dynamic motor parameters a few hundred 
milliseconds (typically 100-1,000 ms) in 
the future. Multiple computational models 
are used to simultaneously extract various 
motor parameters (such as arm position and 
velocity, or hand gripping force) in real time 
from the extracellular activity of frontal and 
parietal cortical neurons. Computational 
models are first trained to predict motor 
parameters from the modulations of neuro- 
nal ensemble activity while animals perform 
motor tasks (typically reaching or grasping 
movements) with their own limbs. As the 
result of this training, the models generate a 
‘transform functiom that matches neuronal 
activity patterns to particular movements. 
Next, the mode of operation is switched to 
‘brain control, in which the time-varying 
outputs of the computational models con- 
trol the movements of an artificial device 
(such as a computer cursor or robot limbs) 
to reproduce the subject’s voluntary motor 
intentions®. 

A somewhat different approach for 
model training implemented in invasive 
BMIs in monkeys" and non-invasive BMIs 
in humans’ is based on a supervised 
adaptive algorithm that does not require 
subjects to perform limb movements, but 
rather adapts the model parameters so 
that the model output approximates ideal 
trajectories. 


Principles of neural ensemble physiology 
The advent of BMI research has advanced 
the field of multi-electrode recordings. Here 
we propose a series of principles of neural 
ensemble physiology (TABLE 1) that have been 
derived from, or validated by, BMI stud- 
jes'6711-142.115-119 Ultimately, these principles 
may be used in the development of new 
neuroprosthetic devices (BOX 1). 


The distributed-coding principle. Multi- 
electrode studies in New World'*1!7118 
and Old World monkeys'”’, rats and 
mice**!”°"!? consistently support the idea 
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that information about single motor param- 
eters is processed within multiple cortical 
areas. BMI studies'” have also revealed that 
real-time predictions of motor parameters 
can be obtained from multiple frontal and 
parietal cortical areas. This widespread rep- 
resentation of motor parameters defines the 
distributed-coding principle’”*****. 

The analysis of neuron-dropping curves 
(NDCs) illustrates this principle well. NDCs 
depict a BMI’s prediction accuracy as a func- 
tion of the number of neurons recorded 
simultaneously during a given experimental 
session. NDCs are computed by first meas- 
uring the entire neuronal population's per- 
formance and then repeating the calculation 
after randomly chosen individual neurons 
are removed (dropped) from the original 
sample. In essence, NDCs measure the size 
of neuronal ensembles needed for a given 
BMI algorithm to achieve a certain level 
of performance. FIGURE 1b,c shows a series of 
NDCs that describe the contribution made 
by populations of neurons, located in dif- 
ferent cortical areas, to the simultaneous 
prediction of multiple time-varying motor 
parameters during operation of a BMI by 
a rhesus monkey. This figure shows how 
the predictions of two such parameters — 
hand position (FIG. 1b) and gripping force!” 
(FIG. 1c) — vary as a function of the size of 
the recorded neuronal population’. 

A widely distributed representation of 
each motor parameter does not necessarily 
mean that equally sized neuronal samples 
obtained from each of these cortical areas 
should yield similar levels of predictions' 
(FIG. 1b,c). For instance, in the example shown 
in FIG. 1, the prediction of hand position was, 
on average, better when randomly sampled 
populations of M1 neurons were used than 


when similar samples of posterior parietal 
cortex (PP) neurons were used. Moreover, 
the difference in prediction performance 
was much smaller between these two corti- 
cal areas when gripping force was used as the 
predicted parameter. However, NDC extrap- 
olation to larger samples’? indicates that, if a 
sufficiently large sample of PP neurons could 
be obtained, neural ensembles from the PP 
could eventually accurately predict both 
hand position and gripping force. Although 
the representation of motor parameters is 
distributed in the cortex, cortical areas none- 
theless show a clear degree of specializa- 
tion (but not in an absolute or strict sense). 
Additionally, modulations in neuronal 
activity in different cortical areas that seem 
to be similar (for example, increases in 
activity during rightward movements) may 
underlie different functions in the corti- 
cal motor programme transmitted to the 
spinal cord. 

The observation of distributed repre- 
sentations of motor parameters obtained 
in BMI studies corresponds well with the 
proposition from previous neurophysiologi- 
cal research that brain areas represent infor- 
mation in a holographic manner, and that 
searching for explicit coding (of force, limb 
displacement or behavioural context) may 
be futile. 


The single-neuron insufficiency principle. 
BMI studies have also revealed that, no mat- 
ter how well tuned a cell is to the behavioural 
task in question, the firing rate of individual 
neurons usually carries only a limited 
amount of information about a given 

motor parameter’'*”. Moreover, the 
contribution of individual neurons to 

the encoding of a given motor parameter 


Table 1 | Principles of neural ensemble physiology 


Principle Explanation 


Distributed coding 


The representation of any behavioural parameter is distributed 


across many brain areas 


Single-neuron insufficiency 
Multitasking 


Mass effect principle 


Single neurons are limited in encoding a given parameter 
A single neuron is informative of several behavioural parameters 


A certain number of neurons in a population is needed for their 


information capacity to stabilize at a sufficiently high value 


Degeneracy principle 
assemblies 


Plasticity 


The same behaviour can be produced by different neuronal 


Neural ensemble function is crucially dependent on the capacity 


to plastically adapt to new behavioural tasks 


Conservation of firing 


Context principle 


The overall firing rates of an ensemble stay constant during the 
learning of a task 


The sensory responses of neural ensembles change according to 


the context of the stimulus 
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tends to vary significantly from minute to 
minute”. Reliably predicting a motor vari- 
able, and achieving accurate and consistent 
operation of a BMI for long periods of time, 
therefore requires simultaneous recording 
from many neurons, and combining their 
collective ensemble firing''*. Incidentally, the 
same single-neuron limitations have been 
observed in the rat somatosensory'**'* and 
gustatory systems**!”?"°, and in the cortico- 
striatal system of wild-type and transgenic 
mice’'. We have called this principle the 
single-neuron insufficiency principle. 

The insufficiency of single-neuron firing 
to precisely reproduce a given behavioural 
output has long been appreciated in studies 
in which averaging of neuronal activity over 
many trials was required to quantify a given 
neurons behavioural function’*!'”. This 
analytical strategy is typically used when 
animals have attained a highly stereotyped 
behavioural performance, after being over- 
trained in a given task. Despite this caveat, 
single neurons have often been attributed 
very specific functions, and their inherent 
noisiness — clearly verified when single tri- 
als are analysed independently — has been 
disregarded’. In such studies, peri-event 
time histograms and directional tuning 
curves have emphasized a consistent rela- 
tionship between the modulations of the 
firing rate of a single cell and behavioural 
parameters. As the attention of neurophysio- 
logical investigations started to shift towards 
ensemble recordings, neuronal variability, 
as opposed to consistency, came into focus, 
and neurophysiologists started to realize that 
modulations in neuronal firing are usually 
highly transient and plastic**'*?"’. This led 
researchers to question the classic assertion 
that behavioural parameters are encoded 
only by the modulation of the firing rate of 
individual cells, and to the realization that 
the precise timing and correlations of neu- 
ral ensemble firing should be taken more 
seriously®**”"**, Usually, in BMIs based on 
recordings from large neuronal popula- 
tions, single-neuron noisiness is removed by 
ensemble averaging. In other words, as the 
population recorded becomes larger, vari- 
ability in single-neuron firing declines in 
importance. 

A recent study'*? documented significant 
single-neuron tuning stability over recording 
sessions that lasted several hours while mon- 
keys performed a reaching task. Although 
this result initially seemed to contradict 
the earlier claim that there is single-neuron 
discharge variability”, these two points 
of view proved to be consistent. The study 
demonstrating tuning stability focused on 
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Box 1| From ensemble principles to neuroprosthetic development 


Ultimately, we expect that the identification of principles of neural ensemble physiology will guide 
the development of a generation of cortical neuroprosthetic devices that can restore full-body 
mobility in patients suffering from devastating levels of paralysis, due either to traumatic or 
degenerative lesions of the nervous system. We believe that such devices should incorporate 
several key design features. First, brain-derived signals should be obtained from multi-electrode 
arrays implanted in the upper- and lower-limb representations of the cortex, preferably in multiple 
cortical areas. Custom-designed microchips (also known as neurochips), chronically implanted in 
the skull, would be used for neural signal-processing tasks. To significantly reduce the risk of 
infection and damage to the cortex, multi-channel wireless technology would transmit neural 
signals to a small, wearable processing unit. Such a unit would run multiple real-time 
computational models designed to optimize the real-time prediction of motor parameters. 
Time-varying, kinematic and dynamic digital motor signals would be used to continuously control 
actuators distributed across the joints of a wearable, whole-body, robotic exoskeleton. High-order 
brain-derived motor commands would then interact with the controllers of local actuators and 
sensors distributed across the exoskeleton. Such interplay between brain-derived and robotic 
control signals, known as shared brain-machine control’*’, would assure both voluntary control 
and stability of bipedal walking of a patient supported by the exoskeleton. 

Touch, position, stretch and force sensors, distributed throughout the exoskeleton, would 
generate a continuous stream of artificial touch and proprioceptive feedback signals to inform 
the patient’s brain of the neuroprosthetic performance. Such signals would be delivered by 
multichannel cortical microstimulation directly into the patient’s somatosensory areas. Our 
prediction is that, after a few weeks, such a continuous stream of somatosensory feedback 
signals, combined with vision, would allow patients to incorporate, through a process of 
experience-dependent cortical plasticity, the whole exoskeleton as an extension of their body. 

These developments are likely to converge into the first reliable, safe and clinically useful cortical 
neuroprosthetic. To accelerate this process and make this milestone a clinical reality, a worldwide 
team of neurophysiologists, computer scientists, engineers, roboticists, neurologists and 
neurosurgeons has been assembled to launch the Walk Again Project, anon-profit, global 
initiative aimed at building the first cortical neuroprosthetic capable of restoring full-body 


mobility in severely paralysed patients. 


mean firing characteristics, obtained by 
averaging hundreds of behavioural trials, to 
extract the preferred movement direction of 
single neurons. However, this study clearly 
showed that the firing rate of a single M1 
neuron varied significantly from trial to trial 
(15-35 spikes per second). Only by averag- 
ing many trials were those authors able to 
obtain smooth directional tuning curves. 
The earlier study'* examined a large popula- 
tion of individual neurons and focused on 
shorter behavioural epochs, during which 
they observed considerable variability. The 
difference between these studies therefore 
resides mainly in the temporal scale and the 
analytical procedure used to estimate short- 
term versus long-term changes in neuronal 
tuning properties. 

In any neuronal population sample there 
are cells that are better tuned to a given 
motor parameter of interest. Such neurons 
are usually called task-related cells”. 
However, even these cells show significant 
variability in their discharges and need to be 
combined to produce accurate predictions of 
motor parameters”. 

Although single cells are generally insuffi- 
cient for obtaining accurate BMI predictions, 
the performance of a single-cell BMI has been 
shown to improve with training”. Indeed, in 


our own studies using large neural ensem- 
bles to drive BMIs, we observed that both 
the firing patterns of individual cells and the 
correlation between cells underwent plastic 
changes that improved BMI accuracy’”'". 


The neuronal multitasking principle. BMI 
experiments also indicate that individual 
neurons, located in each of the corti- 

cal areas sampled, can participate in the 
encoding of more than one parameter at 

a given moment in time’. In other words, 
although individual cortical neurons might 
be better tuned to a given motor parameter, 
they can still contribute simultaneously 

to multiple, transient functional neural 
assemblies and therefore encode several 
motor parameters at once’”*. Here we name 
this the multitasking principle. 

The multitasking principle, described 
here for BMI studies, is similar to the multi- 
modal interactions observed previously in 
sensory and associational cortical areas". 
However, as most of the BMI literature 
deals with the motor system, we prefer to 
use the term ‘multitasking’ In our notation, 
a multitasking BMI controls several motor 
parameters simultaneously, for example 
several degrees of freedom of a multi-joint 
actuator. 
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BMI experiments in which monkeys used 
cortical activity to control the reaching and 
grasping movements of a robotic manipula- 
tor revealed that the firing of single cortical 
neurons was typically correlated to several 
motor variables, such as the manipulator 
position coordinates and its gripping force’. 
Recent experiments that aimed to use the 
combined activity of primate cortical neu- 
rons to reproduce patterns of bipedal loco- 
motion’ revealed that the firing of single 
neurons could contribute to the prediction 
of several motor variables related to leg 
movements!**1°°, including the timing 
of movement onset, as previously observed 


for hand movements!"*. 


The neuronal mass principle. Further analy- 
sis of the NDCs shown in FIG. 1b,c shows 
that parametric reductions in the size of 

the neuronal population initially produce a 
minor reduction in overall prediction per- 
formance for each motor parameter in each 
of the sampled cortical areas!’*”. However, 
below a certain critical population size the 
accuracy of the predictions starts to fall 
more rapidly and, at a certain level (fewer 
than ~10-20 neurons), becomes poor’. 
This suggests that BMIs based on recording 
the activity of just a few neurons are likely 
to perform poorly. According to the single- 
neuron insufficiency principle, predictive 
performance should increase continuously 
as a function of the growth in neural ensem- 
ble size. However, NDCs revealed that when 
the number of neurons used went above a 
certain population size (tens of neurons), the 
amount of predictive information obtained 
tended to remain virtually constant, regard- 
less of the identity of the individual neurons 
sampled. This result is attributable to a sig- 
nificant decrease in the variance of NDCs 
for sufficiently large neuronal samples'"*. 

In other words, once a certain critical 
neuronal mass had been achieved, differ- 
ent, and sufficiently large, random samples 
of single neurons from a given cortical area 
(from different layers or different subregions) 
tended to yield similar levels of predictive 
information about a given motor param- 
eter’. These results led us to propose the 
neuronal mass effect principle, which states 
that to achieve a sufficiently accurate and 
stable prediction of a given motor parameter, 
a neural ensemble has to recruit a crucial 
number of neurons at each moment in time. 
The neuronal mass needed to achieve stabil- 
ity depends on several factors, including the 
presence of highly tuned neurons in the pop- 
ulation'’®'”. If these are missing, predictions 
gradually improve with neuronal sample size, 
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and the noise in the combined population 
activity is proportional to the square root of 
the number of neurons. The critical neuronal 
mass is also highly dependent on neuronal 
correlations, which limit the information that 
the population can contain'*”'**. Correlation 
makes neuronal encoding redundant. As a 
consequence, beyond a certain size the infor- 
mation represented by a neuronal population 
increases only marginally with the addition 
of new cells. 

The minimal size of a neuronal sample 
needed to effectively control a BMI has 
become a controversial issue (for contrast- 
ing opinions, compare REFS 7,11,12 with 
REFS 1,6,9,13,42,116). On the basis of dem- 
onstrations that involved stereotypical and 
relatively simple upper-limb movements, 
several groups have argued that BMIs 
intended to restore upper-limb mobility 
could operate using small neuronal samples 
(<30 neurons)”!"!*!°, Despite this emphasis 
on the role of small neuronal samples, and 
results showing improvement in accuracy of 
small-sample BMIs with training™, practical 
BMIs might not perform sufficiently well 
using signals from only a few neurons, for 
various reasons. For instance, it is unclear 
from the studies that used this approach 
whether small-sample BMIs can sustain the 
same level of performance over long periods 
of time’!*!©. Current recording techniques 
may not allow the sampling of high numbers 
of highly tuned neurons, or provide the 
kind of stability needed for such small- 
sample BMIs to remain effective for 
many months, let alone for many years. 
Additionally, small-sample BMIs may not 
be able to generalize their function to cope 
with newer or more complex behavioural 
tasks'*’. We therefore feel that it is likely that 
such an increase in behavioural demand 
will be met by only large neuronal popula- 
tions. Evidence obtained in our laboratory 
indicates that this is precisely the case for 
BMIs aimed at reproducing both upper- and 
lower-limb movements”’. 


The neural degeneracy principle. BMI stud- 
ies also revealed that a single motor output 
is often associated with distinct spatiotem- 
poral patterns of neural ensemble firing 
on the millisecond scale"*"*'". Following 
the nomenclature introduced by Reeke and 
Edelman!®, this principle, which states that 
identical behavioural outputs can be pro- 
duced by distinct functional and transient 
neural ensembles, has been named the 
degeneracy principle. 

Neural degeneracy is similar to neural 
redundancy in that different combinations 


of single neurons belonging to a neural circuit 
can produce different spatiotemporal fir- 

ing patterns that end up encoding the same 
motor outputs'®. Degenerate coding has 
been demonstrated in several neural circuits, 
including the pyloric network of the lobster, 
the song control system of the zebra finch and 
the order-encoding system of the locust’, 
where it serves to represent low-dimensional 
information by a high-dimensional neural 
network in a fault-tolerant way. 

BMIs based on neuronal ensemble 
recordings solve a similar problem: they map 
the activity of several hundred neurons 
onto the lower number of degrees of 
freedom of an artificial actuator. In these 
experiments, we have observed that similar 
movements, produced either by the animal's 
arm or by an artificial actuator, can result 
from distinct spatiotemporal patterns of 
neuronal population activity’”’. Therefore, 
ifa sufficiently large population of neurons 
is recorded simultaneously, movements 
induced by a BMI can be reliably produced in 
each behavioural trial. Similarly, we observed 
that stereotypical steps in bipedally walking 
monkeys were associated with different pat- 
terns of motor cortex activations’»’. It follows 
from these considerations that the basic pro- 
portion between the recorded ensemble size 
and the number of controlled degrees of free- 
dom should be preserved for BMI applica- 
tions that require the production of complex 
motor behaviours in artificial actuators. 


The plasticity principle. Experience- 
dependent plasticity in cortical neural 
ensembles!*”'® is essential for primates to 
learn to operate a BMI. As mentioned above, 
the strength of a single-neuron correla- 
tion to a given motor parameter is typically 
imprecise, varying as a function of time, 
internal state and learning, as well as the 
animal’s expectation of the task outcome 
and reward''*!?5'®, Several studies have 
now documented the occurrence of corti- 
cal plasticity as animals learn to operate a 
BMI"!***, This phenomenon is character- 
ized by changes in the tuning properties of 
individual neurons!” and physiological 
adaptations at the level of neural ensembles, 
which include changes in firing covariance 
and spike timing'. Such changes in neuronal 
properties are undoubtedly related to basic 
plasticity mechanisms, such as changes in 
the strength of synaptic connections and 
gene expression. However, in BMI experi- 
ments such basic mechanisms are difficult 
to isolate from the population effects. For 
example, increases in the firing activity of 

a given neuron can result from multiple 


factors, such as changes in synaptic strength, 
increases in excitatory inputs or release of 
inhibition. Combinations of such factors 
manifest themselves as changes in neuronal 
tuning (the correlation of cell firing with a 
given motor parameter). In BMI studies, 
similar measurements of neuronal tuning 
are provided by the weights that prediction 
models assign to different neurons'” and 
time-dependent correlations of neuronal 
rates with kinematic parameters”. 

As a rule, neuronal tuning tends to be 
modified and refined as a result of operant 
conditioning****!5!!7°"'”, In BMI studies, 
cortical plasticity manifests itself in a series 
of physiological adaptations. For instance, 
during the transition from manual to brain 
control of a BMI'” (when animals ceased 
to use their own limbs and started to con- 
trol an actuator using their cortical activ- 
ity directly), a significant portion of the 
recorded neurons, which were distributed 
across multiple cortical areas, progressively 
acquired tuning properties related to the 
kinematic properties of the robotic device 
used (FIG. 2). Asa result, a fraction of these 
cortical neurons showed tuning to the 
kinematic properties of both the animal's 
biological arms and the robotic arm (FIG. 2a). 
Conversely, a subset of the recorded cortical 
neurons ceased to fire, or to show velocity 
or direction tuning, when animals stopped 
producing arm movements and controlled 
a robotic device without any overt motor 
behaviour!” (FIG. 2b). Perhaps more surpris- 
ingly, a fraction of the recorded cortical 
neurons showed clear velocity and direction 
tuning that was related to the movements 
of the robotic prosthesis but not to the 
displacement of the animal’s own arms!” 
(FIG. 2c). Such tuning developed and became 
sharper during the period in which monkeys 
learned to operate the BMI without execu- 
tion of overt body movements (brain control 
mode). The emergence of such tuning may 
explain why monkeys were able to control 
both robotic arms and legs using BMIs 
without generating corresponding body 
movements. 

Besides changes in single-neuron tuning 
properties, a significant increase in firing 
covariance between pairs of neurons, located 
within and between multiple cortical areas, 
has also been observed when animals started 
operating a BMI without moving their own 
limbs! (FIG. 2d). As animals shifted back and 
forth between using their own limbs or the 
artificial actuator controlled by the BMI to 
solve a particular motor task, functional 
coupling between pairs of cortical neurons 
adapted dynamically. Interestingly, this 
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increase in neuronal pair covariance was 
observed not only within a given cortical 
area, but also between neurons located in 
distinct cortical fields’. 

The observation of such a broad rep- 
ertoire of functional cortical adaptations 
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Figure 2 | Neuronal activity during a reaching task. The task illustrated 
was performed by a rhesus macaque that controlled a robotic actuator 
using a hand control or through a brain-machine interface (BMI; brain con- 
trol). During BMI operation, the monkey either continued to move the pole 
with the hand (brain control with hand movements) or stopped moving its 
hand (brain control without hand movements). a | Activity of a primary 
motor cortex (M1) neuron during both pole and brain control. Colour-coded 
diagrams represent neuronal tuning to movement velocity (that is, the aver- 
age neuronal rate as a function of hand or robot velocity), calculated at dif- 
ferent lags (-400 ms to +500 ms) with respect to the time of velocity 
measurement. The diagrams labelled ‘Hand’ represent neuronal tuning to 
hand movements, and the diagrams labelled ‘Robot’ represent tuning 
to robot movements. During brain control without hand movements, this 
neuron became less tuned to hand movements (row of colour diagrams 


during the operation of BMIs supports 
many far-reaching conclusions. First, they 
suggest that Old World monkeys may be 
capable of ‘motor imagery’*’"**: to imagine, 
in great detail, a series of complex motor 
sequences without necessarily producing 
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body movements to execute such motor 
plans. Second, they imply that, at its limit, 
cortical plasticity may allow artificial tools 
to be incorporated as part of the multiple 
functional representations of the body that 
exist in the mammalian brain. If this proves 
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labelled ‘Brain control with hand movements: hand’). Tuning to robot move- 
ments was maximal during brain control without hand movements (row 
labelled ‘Brain control without hand movements: robot’). b | An M1 neuron 
modulated only when the monkey moved its hand. ¢ | An M1 neuron that 
was not modulated during hand movements, but became tuned to the robot 
movements during brain control without hand movements. d | Analysis of 
pairwise correlations in firing between the neurons in the recorded ensem- 
ble, using data from REF. 1. Correlations increased during brain control, 
especially brain control without hand movements. The highest correlations 
were between the neurons recorded in the same cortical area. M1ips, 
primary motor cortex, hemisphere ipsilateral to the working hand; PMd, 
dorsal premotor cortex; S1, primary somatosensory cortex; SMA, supple- 
mentary motor area. Images in parts a—c are modified, with permission, from 
REF. 42 © (2005) Society for Neuroscience. 
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to be true, we would predict that continu- 
ous use of a BMI should induce subjects to 
perceive artificial prosthetic devices, such 
as prosthetic arms and legs, controlled by 
a BML as part of their own bodies. Such a 
prediction opens the intriguing possibility 
that the representation of self does not nec- 
essarily end at the limit of the body surface, 
but can be extended to incorporate artificial 
tools under the control of the subject’s brain. 
BMI research further stretches this puzzling 
idea by demonstrating that, once brain activ- 
ity is recorded and decoded efficiently in real 
time, its capacity to control artificial devices 
can undergo considerable modification in 
terms of temporal, spatial, kinematic and 
kinetic characteristics, termed scaling’. 
In other words, not only can a BMI enact 
voluntary motor outputs faster than the sub- 
ject’s biological apparatus (temporal scaling), 
but it can also accomplish motor tasks at a 
distance from the subject’s own body (spatial 
scaling), by controlling an actuator that is 
either considerably smaller (for example, a 
nano-tool) or considerably larger (for exam- 
ple, a crane) than the subject’s own biological 
appendices. 

Recently, another powerful way to induce 
cortical plasticity has been introduced 
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Figure 3 | Discrimination of spatiotemporal microstimulation patterns 
by owl monkeys. Microstimulation trains were delivered to the primary 
somatosensory cortex through chronically implanted multi-electrode 
arrays. The monkeys responded to microstimulation or its absence by select- 
ing the target of reaching movements. Top panels illustrate microstimula- 
tion patterns; bottom panels show discrimination accuracy as the function 
of training day. a | A basic task in which the monkeys detected the presence 
or absence of microstimulation. The monkeys learned the task in 1 month. 


to BMI research: multichannel, cortical 
microstimulation'>!™*. FIGURE 3 shows some 
of the findings obtained when chronic 
multichannel microstimulation of the 
primary somatosensory cortex was used 

to instruct owl monkeys on how to locate 
food rewards. During several months of 
microstimulation sessions, these monkeys 
progressively learned to detect the pres- 
ence or absence of microstimulation, and 

to discriminate different temporal patterns 
of microstimulation pulses that indicated 
food location''®. Moreover, the animals also 
learned new behavioural contingencies 
after changes were made in the direction 

of arm reach instructed by microstimula- 
tion. Interestingly, monkeys required less 
time to master a new set of rules as train- 
ing progressed and new task contingencies 
were introduced, which allowed them more 
practice in handling microstimulation cues 
(FIG. 3b,c). So after being exposed to an origi- 
nal basic rule, monkeys learned a reversed 
task much more rapidly and, subsequently, 
more elaborate contingencies as well". 
Although the basic mechanism involved in 
such ‘rule generalization was not uncovered, 
these results confirm the hypothesis that 
functional plasticity of cortical tissue can 
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be induced by intracortical microstimula- 
tion. This raises the question of whether 
chronic cortical microstimulation can trig- 
ger a process of functional adaptation that 
leads to the emergence of realistic perceptual 
experiences. Although there is no defini- 
tive answer to this question, it is interesting 
to note that people who were exposed to 
chronic patterned cutaneous stimulation, as 
an artificial replacement strategy for vision, 
learned to use such artificial sensory input 
to guide their movements and reported the 
development of qualitatively new percep- 
tions'**'8°, Confirmation of new perceptual 
experiences after prolonged training with 
microstimulation would certainly be of con- 
siderable relevance for the design of future 
neuroprosthetic devices that aim to restore 
upper- and lower-limb mobility in severely 
paralysed patients. 

With that long-term vision in mind, 
we have recently started to develop a new 
paradigm, named brain-machine-brain 
interface (FIG. 4), that will enable us to test 
whether monkeys can use neural ensemble 
activity to control the movements of artifi- 
cial devices guided by instructions delivered 
directly to their somatosensory cortices by 
multichannel microstimulation. 


¢ Spatiotemporal discrimination task 
150 ms 
—— 
Electrode pair 1 (EP 1) 


EP 2 
EP 3 
EP 4 


Versus 
EP 1 


EP 2 
EP 3 
EP 4 


Fraction correct 


0.60 
0.555 
0.50 


Chance level 
rT TT 
6 7 8 9 10 
Experiment 


rT TT 
2 3 4 °5 


b| Discrimination of temporal patterns of microstimulation. The monkeys 
learned the task in 1 week. c | A spatiotemporal discrimination task during 
which waves of microstimulation were delivered through the electrode 
arrays. The monkeys learned the task in 3 days, and could then discriminate 
spatiotemporal patterns of cortical microstimulation. Moreover, after pro- 
longed training with microstimulation they learned to interpret new micro- 
stimulation patterns faster. Figure is reproduced, with permission, from 
REF. 115 © (2007) Society for Neuroscience. 
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b Electrode array 


PERSPECTIVES 
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Output 


Figure 4 | The concept of a brain-machine-brain (BMBI) interface with 
artificial sensory feedback. In one possible implementation, depending on 
the presence or absence of microstimulation the monkeys perform brain— 
machine interface (BMl)-controlled reaching movements in different directions 
(right or left). Initially the monkeys acquire visual targets using a screen cursor 
moved by a hand-held joystick, and the directional instruction is delivered by 
mechanical vibration of the joystick handle. Manual control is then replaced 
by BMI control of cursor movements, and vibration is replaced by cortical 
microstimulation. a | Examples of possible sites of cortical implantation. 


communication | 


Signal Decode 
——_—_—_—_—_——— | ——_—_—_—__—_—-_-_-_ —- 
processing neural data 
Visual feedback | 
Real-time 
task control 


Target selection task 


Input 


communication | 


Multi-electrode arrays are placed in the dorsal premotor cortex (PMd), the 
primary motor cortex (M1) and the primary somatosensory cortex (S1). PMd 
and M1 arrays are used to extract motor commands, and the S1 is the site of 
microstimulation. b | Examples of possible locations of S1 electrodes with 
respect to a somatotopic map determined using receptive field measurements. 
The multi-electrode array covers the representation of digits D2-D5 and of the 
hand pads. c | Receptive fields of the electrodes through which micro- 
stimulation is delivered. d | Parameters of microstimulation train (top) and 
microstimulation pulses. e | Schematic of the experiment, as described above. 


The conservation of firing principle. Despite 
documenting clear and widespread changes 
in the single-neuron firing rate related to 
plastic modifications in neuronal velocity 
and duration tuning, and increases in firing 
covariance between pairs of cortical neu- 
rons, we have also observed that the global 
firing rate (total number of spikes) of the 
cortical neural ensembles recorded in our 
experiments usually remained unchanged 

as animals learned to operate a BMI’. This 
principle of neural ensemble firing conserva- 
tion has also been observed in various other 
studies — including experiments conducted 
in New World monkeys, rats and mice — 
involving distinct cortical areas and various 
motor and sensory tasks’*!7°1'87-, These 
studies indicate that maintaining the total 


number of spikes for a range of behaviours 
could be a pervasive, homeostatis-like 
mechanism of cortical ensembles. 


The context principle. Multi-electrode 
recordings in freely behaving animals have 
also opened new ways to examine a funda- 
mental question in classic neurophysiology: 
how neurons respond to sensory stimuli that 
are applied passively or acquired actively by 
subjects. A study in behaving rats trained to 
perform a tactile discrimination task using 
only their facial whiskers addressed this 
issue directly’®. This study revealed that 
neuronal modulations evoked by passively 
versus actively acquired tactile stimulation 
were strikingly different in their magnitude, 
adaptation rate and percentage of excitatory 


versus inhibitory sensory evoked responses 
in the primary somatosensory cortex (FIG. 5). 
A similar result has since been described in 
the rat primary gustatory cortex'”’ and in the 
auditory cortex of marmosets”. 

Such marked neurophysiological differ- 
ences indicate that the context in which ani- 
mals sample their surrounding environment 
can radically alter the way cortical neural 
ensembles respond to incoming sensory 
information. Therefore, we named this 
principle the context principle. 


Conclusions 

The principles of neural ensemble physiol- 
ogy described above were either derived 
from, or confirmed by, a decade of BMI 
experiments. This demonstrates that, in 
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Figure 5 | Neuronal responses in rat somatosensory cortex to passively 
applied stimuli versus active discrimination of the same stimuli. 
a| Schematics illustrating the whiskers stimulated (arranged in rows and 
columns) and the stimulus timing. Multi-whisker ramp-and-hold stimuli 
were delivered to anaesthetized (top) or awake restrained (bottom) rats. 
Large circles represent stimulation of a particular whisker. Arrows show 
stimulation onsets. b | Schematic of stimulus delivery. The aperture was 
accelerated across the facial whiskers by the pneumatic solenoid and also 
simultaneously deflected laterally in varying amounts by the direct-current 


servo to accurately replicate the range of whisker deflection dynamics that 
occurred during active discrimination. c | Mean duration and magnitude 
of the responses evoked during active discrimination and during 
delivery of passive stimuli to anaesthetized or awake restrained rats. These 
results indicated that neuronal responses evoked in the primary somato- 
sensory cortex by passively applied stimuli were strikingly different from 
those evoked by actively acquired tactile stimulation. Figure is reproduced, 
with permission, from REF. 169 © (2004) American Association for the 
Advancement of Science. 
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Neuroprosthetic devices based on brain-machine interface technology hold promise for the restoration of body 
mobility in patients suffering from devastating motor deficits caused by brain injury, neurologic diseases and limb 
loss. During the last decade, considerable progress has been achieved in this multidisciplinary research, mainly in the 
brain-machine interface that enacts upper-limb functionality. However, a considerable number of problems need to 
be resolved before fully functional limb neuroprostheses can be built. To move towards developing neuroprosthetic 
devices for humans, brain-machine interface research has to address a number of issues related to improving the 
quality of neuronal recordings, achieving stable, long-term performance, and extending the brain-machine 
interface approach to a broad range of motor and sensory functions. Here, we review the future steps that are part 
of the strategic plan of the Duke University Center for Neuroengineering, and its partners, the Brazilian National 
Institute of Brain-Machine Interfaces and the Ecole Polytechnique Fédérale de Lausanne (EPFL) Center for 
Neuroprosthetics, to bring this new technology to clinical fruition. 
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INTRODUCTION 


Millions of people worldwide suffer from sensorimotor 
deficits caused by neurologic injuries, diseases or limb loss. 
According to recent data reported in Medical News Today, 
five million people in the USA alone currently suffer from 
some type of severe body paralysis.’ Currently, there is no 
cure for such devastating cases of paralysis, for example 
complete spinal cord injury (SCI).” Meanwhile, treatment is 
only partially effective in less severe cases.’ Neural 
prosthetic devices based on brain-machine interfaces 
(BMIs) hold promise to restore both partial and full body 
mobility in paralyzed patients.*’° BMIs bypass the site of 
the neural lesion and connect the remaining healthy motor 
areas of the brain, particularly the motor cortex, directly to 
assistive and prosthetic devices that can take the shape of, 
for example, robotic limbs or a full body exoskeleton. The 
main idea behind BMIs is to employ the activity of healthy 
motor brain areas, which in many cases of paralysis remain 
capable of generating motor commands despite being 
disconnected from the body effectors,'’ to control artificial 
tools that restore the patient’s mobility. 


Copyright © 2011 CLINICS - This is an Open Access article distributed under 
the terms of the Creative Commons Attribution Non-Commercial License (http:// 
creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted non- 
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During the last decade, the field of BMIs has experienced 
an explosive development.”” Hence, it has generated high 
expectations among neuroscientists, physicians and patients 
alike, regarding its potential clinical applications. A number 
of BMI systems have been studied in rodents’? and 
nonhuman primates.'*'7 BMI technology also entered 
human clinical research where both non-invasive EEG- 
based systems”'®'? and invasive BMIs based on brain 
implants~’** have been tested. Notwithstanding the success 
of these pioneering experiments, a number of issues need to 
be resolved before a fully functional practical neuropros- 
thetic for long-term use can be built.” These include: implant 
biocompatibility issues;** increasing the number of neural 
channels of the recording system; improving BMI decoding 
algorithms; building fully implantable systems; sensorizing 
neuroprosthetic limbs; and extending the BMI approach to a 
broader range of motor control tasks, especially tasks that 
require lower limb control: bipedal walking~* and upright 
posture control.” 

The Duke University Center for Neuroengineering 
(DUCN) has been at the forefront of BMI research on cortical 
prosthetic devices for motor rehabilitation since this field 
emerged about 12 years ago. At the DUCN, we have 
developed pioneering BMI systems that enact a wide range 
of motor functions, from arm reaching and grasping 131726 465 
bipedal locomotion™”” in a variety of artificial actuators. 
DUCN researchers were also the first to incorporate artificial 
somatic sensation in BMIs.”°?? Here, we review the most 
recent findings of the BMI initiative at the DUCN and discuss 
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its perspectives and strategic plan for the near- and long-term 
future of BMI research. 


BMI COMPONENTS 


The essential components of a BMI system are well 
captured by the BMI that enacts reaching and grasp- 
ing’ ee In this BMI design, a rhesus monkey controls 
a robotic arm with its motor cortical activity, while visual 
and/or somatosensory feedback signals from the robot are 
delivered back to the brain as either natural visual stimuli 
or, in the case of artificial tactile information, intracortical 
microstimulation (ICMS) of the primary somatosensory 
cortex ($1).”7%° In these studies, we implant multielectrode 
arrays in multiple cortical areas of the rhesus monkey’s 
brain.*” The present generation of chronic multielectrode 
implants is capable of recording the extracellular electrical 
activity of hundreds of cortical cells.'°**7°°° As a result of 
recent technological developments, this benchmark number 
is expected to rise to several thousands of simultaneously 
recorded neurons in the next decade. Recording such large- 
scale neuronal ensemble activity is crucially important for 
BMI accuracy.” Concurrent neuronal ensemble activity is 
processed by BMI decoding algorithms which translate 
myriad neuronal spikes into continuous signals that drive 
the robotic arm’s movements, according to the voluntary 
motor intentions of the subject. The BMI setup also includes 
the data acquisition system, the computer cluster running 
multiple decoding models in real time, the robot arm, the 
visual display and a sensory feedback loop from the 
actuator to the brain. Below we discuss these key BMI 
components in more detail. 


LARGE SCALE NEURONAL RECORDINGS 


Our work on BMIs has clearly demonstrated that a large 
number of recording channels is needed for accurate 
extraction of motor intentions from the brain.?1817912 
Shifting from the classic focus on single neurons, today, 
more and more evidence accumulates in favor of the notion 
that distributed ensembles of neurons define the true 
physiologic unit of the mammalian central nervous system.” 
During the last two decades, advanced electrophysiological 
methods have allowed recording from progressively larger 
samples of single neurons in behaving animals.*”***? This 
methodology has equipped neuroscientists with better tools 
to study the neurophysiologic principles that define the 
operation of the cortex,” and has also made the idea of BMIs 
practical. Today, the most advanced BMIs developed at the 
DUCN utilize simultaneous recordings of the extracellular 
electrical activity of hundreds of individual neurons.*° 

Neuron-dropping analysis, a technique developed at the 
DUCN to measure the dependence of parameter extraction 
accuracy on the number of simultaneously recorded 
neurons,’” shows that BMI accuracy increases with the size 
of the neuronal ensemble. Typically, the best extractions of 
motor parameters are obtained when the activity of 
populations of neurons from several cortical areas is 
recorded simultaneously.'*** Importantly, the number of 
required neurons increases as more motor parameters are 
simultaneously extracted.™* 

Large neural ensembles confer redundancy of control 
and, hence, reliability.’ In our current research on BMIs that 
enact arm and leg movements, we can obtain a peak 
signal-to-noise ratio (SNR) of 10dB using approximately 100 
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task related neurons.”**° This corresponds to a 5 bits/s 
bandwidth. SNR can be further improved via time 
averaging, but this slows down the movements. Realistic 
and useful movements should have a SNR of about 20 to 
30dB. Indeed, we consider this benchmark as the lower 
bound that one should aim in order to build a neuropros- 
thetic device for clinical use. Thus, to estimate the number of 
neurons needed to obtain this level of accuracy, we assume 
that noise decreases as the inverse of the square root of the 
number of neurons—an assumption that holds for both 
Gaussian and Poisson noise sources. Under this assumption, 
a tenfold increase in the sample of neurons recorded 
simultaneously would be needed to achieve the 10dB of 
improvement needed. Therefore, to obtain a control signal 
with 20dB fidelity (the DUCN short-term goal), recordings 
involving 1000 neurons are needed; to obtain 40dB (the 
DUCN long-term goal), 100,000 neurons are needed. In 
addition, it should be emphasized that commensurately 
more neurons are required for prosthetic limbs with many 
degrees of freedom (DOFs). As practical neuroprosthetics 
for paralyzed subjects will have to cope with at least 4-10 
DOF, demands for large samples of recorded neurons will 
remain a central bottleneck for the development of clinical 
applications of BMIs for the foreseeable future. Thus, only 
when a new generation of high-channel count recordings 
becomes available can practical neuroprosthetics for clinical 
use be implemented with a reasonable expectation for 
clinical success. At the DUCN, we expect to conclude the 
development of such new technology and, as a conse- 
quence, launch the testing phase of our first clinical 
application of a BMI for restoring full body mobility by 
the summer of 2014. 

To achieve this goal, we have been developing advanced 
sensors that sample large-scale extracellular electrical 
activity from the brain. In our most recent design of 
multielectrode implants, electrodes are arranged into sub- 
sets sitting inside guiding tubes.” The electrodes within 
these subsets have different lengths. The guiding tubes are 
spaced at 0.5-2 mm, and each of them is independently 
movable. This new three-dimensional (3D) design, named 
‘the multi-electrode recording cube’ not only improves the 
size of the potential neuronal sample recorded simulta- 
neously per probe, but also enhances the implantation 
capacity of the microelectrodes in the brain tissue as each 
electrode subset penetrates individually, minimizing corti- 
cal dimpling. Currently, we are expanding this new cube 
design to develop the next generation of implants that will 
increase the number of potential active recording sites per 
cube to about 1500. Also, as the size of each implanted 
microelectrode is of vital importance when so many 
electrodes are inserted into richly vascularized brain tissue, 
it is imperative to produce as little tissue displacement as 
possible during the surgical implantation of these recording 
devices. This issue will be resolved by removing the 
structural elements after implantation, thereby freeing the 
microelectrodes made of smaller diameter microwires than 
that required to pierce and penetrate the brain tissue. Not 
only will small diameter microwires minimize the mechan- 
ical distortion of nervous tissue, it will also help to minimize 
microglial and other immune responses to the foreign 
material. Our approach will allow implantation of small- 
diameter wires and thereby avoid failures associated with 
large-diameter implants. Fine electrodes will be guided into 
the brain with a strong, stiff tungsten central shaft. Later on, 
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the shaft will be removed, leaving the electrodes in the 
brain. Each single electrode shaft will carry 10-20 recording 
microwires, staggered to cover the targeted nervous tissue. 
To refer properly to this new technology and to distinguish 
it from previous approaches, we have coined the term ‘very- 
large scale multichannel brain implants’ (VLS-MBI). 


DECODING ALGORITHMS 


Large-scale neuronal activity recorded by our very large- 
scale multichannel brain implants will be processed by real- 
time BMI encoding algorithms that extract behavioral 
parameters, for example kinematic parameters of many 
DOF limb movements. At the DUCN, we employ an 
integrated BMI suite that combines the neurophysiologic 
recording and stimulation hardware, as well as the 
computer cluster that runs behavioral tasks and BMI 
decoders.*”” We have incorporated into this BMI suite 
several neuronal decoders: the unscented Kalman filter,*° 
the Wiener filter, artificial neural networks, and discrete 
state Bayesian approaches.** In particular, the unscented 
Kalman filter makes Bayesian inferences of the repetitive 
patterns of the movements performed during arm reaching 
tasks, as such patterns will occur frequently in the tasks that 
a practical neuroprosthetic limb has to perform. For 
example, a prosthetic limb that aids in feeding moves back 
and forth between the food item and the user’s mouth. The 
unscented Kalman filter exploits non-linear models of 
neural tuning and prior knowledge about movement 
patterns. In addition, it keeps a short history of the state 
variables—in this case, the desired limb movements. This 
algorithm also captures complex patterns in the desired 
movements. At the DUCN, the unscented Kalman filter was 
tested in a BMI that enacted arm reaching movements and 
achieved significantly better accuracy as compared to the 
Kalman filter, the Wiener filter, the population vector 
method, and the stochastic state point process filter.°® 

In addition to the unscented Kalman filter, we employ a 
multiple-model-switching paradigm.”* In this approach, 
separate submodels are trained to decode particular 
behavioral states (e.g. the grasping phase of the reach and 
grasp movement versus the reaching phase, or walking 
forward versus walking backward), and the state predictor 
model serves to detect the state and select the appropriate 
submodel. The simplest switching model works as a 
combination of three linear decoding models: a model for 
predicting state 1 (e.g. reaching), a model for state 2 (e.g. 
grasping) and the paradigm predictor model (the switch). 
These models are arranged in two layers with the paradigm 
predictor model controlling a toggle between the two 
kinematic submodels, which are then shunted to the final 
output of the switching model. When it is determined that 
the monkey is performing in state 1, one submodel is used 
to produce the output, and when state 2 is detected, the 
other submodel is used. 

These algorithmic tools allow us to test different neural 
control modes. In the most straightforward implementation, 
we predict the kinematics of each joint of the limb and 
convert them into the actuator configuration. This control 
model is of interest for several reasons. First, this mode of 
operation is similar to the operation of the nervous system in 
controlling natural movements. Second, the number of 
degrees of freedom that can be achieved by a BMI is of great 
interest to those working in this field. At the same time, 
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practical BMI applications will be inconvenient to use if the 
user is required to control each degree of freedom indepen- 
dently. To overcome this problem, we intend to implement in 
our clinical applications a mode of BMI operation in which 
the control over an external actuator is shared between the 
subject’s brain activity and robotic controls. This mode of 
operation is known as shared control of a BMI.” During 
shared control, the subject’s brain is primarily in charge of 
high-order control of movements (when to initiate movement 
and where to move), whereas the low-level coordination of 
the movement is performed by an autonomous controller. 
Such sharing optimizes BMI performance: the user can 
control it through their voluntary intentions, whereas the 
robotic controller assures accuracy and stability. 


BRAIN-MACHINE-BRAIN INTERFACE 


The DUCN research team was the first to add an artificial 
somatosensory feedback loop to a BMI for arm reaching. In 
our initial studies, we used spatiotemporal patterns of ICMS 
to instruct primate arm reaches. In a study conducted in owl 


monkeys,” we addressed two issues that are critical for 
using ICMS as an artificial sensory channel in neural 
prostheses: (i) whether such artificial sensation can be 
evoked by multichannel ICMS; and (ii) whether ICMS is 
suitable for long-term usage in a BMI-like application. We 
explored the first issue by testing the capacity of owl 
monkeys to discriminate multichannel ICMS of increasing 
complexity. We investigated the second issue by testing the 
long-term efficacy of ICMS over many months. We 
discovered that owl monkeys could learn to discriminate 
spatiotemporal patterns of ICMS delivered directly to their 
S1 and guide their reaching movements based on this 
discrimination. Owl monkeys were implanted with multi- 
electrode arrays in several cortical areas, and 51 implants 
were employed to deliver spatiotemporal patterns of ICMS. 
The behavioral task performed by the owl monkeys 
progressed from a simple requirement of detecting the 
presence of ICMS, to the requirement of discriminating 
spatiotemporal patterns created using four electrode pairs. 
The monkeys successfully learned these tasks. Interestingly, 
they learned new microstimulation patterns more rapidly 
compared with initial training. This result suggests that the 
monkey brain may have mimicked the operation of this 
intra-cortical microstimulation paradigm as if it were a new 
sensory channel. 

In a study conducted in rhesus monkeys, we employed 
ICMS to cue monkeys that performed BMI reaching tasks.” 
The behavioral tasks consisted of acquiring visual targets 
with a computer cursor. The monkeys first performed the 
behavioral tasks manually, using a hand-held joystick, and 
later controlled the cursor movements directly with the 
electrical activity of a sample of cortical neurons. Manual 
performance data were used to train linear decoding models 
that extracted cursor position from the modulations of 
populations of simultaneously recorded cortical cells. Once 
the model parameters were calculated, the mode of 
operation was switched to brain control during which the 
joystick was disconnected from the cursor and cursor 
position became directly controlled by the signals extracted 
from the animal’s brain. ICMS of S1 was then used to cue 
monkeys to which direction they had to move their arms 
during execution of a target choice task. In this experiment, 
monkeys had to choose among two visually identical targets 
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based on an instruction, in the format of spatiotemporal 
pattern of ICMS, delivered directly to the animals’ S1. The 
monkeys were implanted with multiple microwire arrays in 
several cortical areas. The dorsal premotor cortex (PMd) and 
primary motor cortex (M1) arrays were used to extract 
motor commands, while the primary somatosensory cortex 
(S1) array was employed as the main target for ICMS. The 
electrodes chosen for ICMS yielded recorded S1 neurons 
with clear receptive fields located on the ventral aspect of 
the second, third and fourth digits and palm pads. Biphasic 
current pulses were injected into S1 through these electrode 
pairs synchronously at 30 to 60 Hz. We simultaneously 
recorded the electrical activity of 50-200 cortical neurons. 
The monkeys learned to perform in BMI control with and 
without using the joystick. 

Moreover, in our recent study,?8 ICMS served as an 
artificial sense of active touch as it conveyed to the monkeys 
the properties of virtual objects that the actuator (computer 
cursor or a virtual image of a monkey arm) touched. Two 
monkeys learned to operate this new apparatus, without 
any need to move their own limbs or use visual feedback to 
solve the task. 

Thus, as a result of converging principles of motor and 
sensory neuroprosthetics, we have demonstrated the feasi- 
bility of moving from a BMI to a brain-machine-brain 
interface in which artificial actuators and brain tissue are 
connected bi-directionally, without any meaningful inter- 
ference or constraint imposed by the physical limits of the 
subject’s body. 


POSITION SENSE 


Having created a paradigm for introducing an artificial 
sense of touch to our BMIs, the next goal of the DUCN is to 
develop neuroprosthetic limbs that incorporate an artificial 
sense of position. Position sense is of great importance for 
clinical applications because ideal prosthetic limbs should 
be perceived as natural extensions of the users’ bodies. 
Normally, such positional signals are provided by the 
afferents of muscles, joints and skin. Afferent information 
ascends to the brain where it is eventually transformed into 
representations encoding limb position in different coordi- 
nate frames, such as body-centered and external space- 
centered coordinates.*’** The complexity of spatial proces- 
sing in the brain makes the task of creating an artificial 
position sense particularly difficult. Theoretically, artificial 
proprioception could be implemented in a straightforward 
way by applying microstimulation to the somatosensory 
cortex or thalamus that reproduces individual joint angles of 
an artificial arm. The subject would then experience 
sensation of many DOFs of the artificial arm and would 
be able to perceive its spatial orientation. However, practical 
realization of such position sense would be problematic 
because of numerous uncertainties in choosing the trans- 
formation from the stimulation patterns to the joint angles. 
Given the complexity of cortical processing of propriocep- 
tive information,’ it would be impossible to implement 
such transformation as a precise mapping from the arm 
joints to the brain somatosensory map. It is also question- 
able if the user would be able to transform the stimulation 
patterns designed to mimic individual joints into a coherent 
position sense. Additionally, certain centrally generated 
components of normal position sense, such as corollary 
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discharges,;* would be difficult to incorporate in such 
implementation. 

Because of these foreseen difficulties in the straightfor- 
ward implementation of an artificial position sense, we 
chose a simpler and more feasible approach in which 
stimulation of S1 is not initially coupled to the orientation of 
the limb position, but instead represents 3D spatial locations 
to which the subject is required to reach. In this approach, 
the subject starts with learning how ICMS of cortical 
somatotopic representations of the body is mapped to a 
3D space. Once the learning of such mapping is perfected, 
the same ICMS pattern is used to represent the spatial 
location of the endpoint of a prosthetic limb, fulfilling the 
goal of providing a limb neuroprosthetic with a position 
sense. This experimental design bears similarity to the 
studies on sensory substitution in which visual information 
was conveyed by the stimulation of skin surfaces,°” with 
the difference that, instead of using peripheral receptors as 
the entry point for the artificial sensory channel, we opted to 
deliver the information directly to the somatosensory cortex 
or thalamus. We expect that similarly to sensory substitu- 
tion using peripheral stimulation, training with stimulation 
of the somatosensory areas of the brain would eventually 
give rise to an artificial position sense that represents 
external space. Furthermore, we expect that as such position 
sense becomes coupled to the position of a virtual image of a 
monkey arm (monkey avatar), monkeys will develop an 
artificial proprioception sense capable of guiding their 
movements thereafter. 


OPTOGENETICS 


Our initial approach to sensorizing neuroprosthetic limbs 
was based on ICMS, which involves delivering small pulses 
of current through microelectrodes directly into the sensory 
areas of the brain.*®°*° Although this technique proved to 
be efficient in bi-directional BMIs, it has relatively low 
spatial resolution*® and produces electrical artifacts that 
saturate neural recording channels,*” causing problems for 
extracting neural activity both during and after this period, 
as it typically occludes 5-10 ms of neural data per pulse. 
ICMS acts on both fibers and neurons, and neurons in the 
stimulated area may become inhibited instead of being 
excited.”’ A further problem is that ICMS pulses have to be 
precisely balanced, otherwise an unbalanced current injec- 
tion could cause damage to the neural tissue stimulated by 
this procedure. 

To explore an alternative method for sensorizing neuro- 
prosthetics, we have started to design a cortical multi- 
channel stimulator based on optogenetics—a new technique 
for the stimulation of neurons. Optogenetics is based 
on genetically modified ion channels that respond directly 
to light.>? These light-gated ion channels, such as 
Channelrhodopsin-2 (Chr-2), allow precise, millisecond 
control of specific neurons.°*°* This technique eliminates 
most of the key problems associated with ICMS: there is no 
associated electrical artifact to interfere with the electro- 
physiological recordings, nor any tissue damage from the 
current injection. It also allows for finer control of the spatial 
pattern of activation. We expect that multichannel optoge- 
netic stimulation combined with multichannel neuronal 
recordings will allow us to develop an artificial position 
sense in monkeys and to study the neuronal mechanisms 
involved in it. Bi-directional BMIs based on such a design 


319 


CLINICS 2011;66(S1):25-32 


would be superior to current designs in both the specificity 
and long-term performance of the sensory loop and the 
quality of neuronal recordings. We are currently building a 
prototype optogenetic implant that will simultaneously 
sample the activity of cortical neuronal ensembles and 
deliver complex stimulation patterns through multichannel 
optogenetic stimulation of cortical sensory areas. Our 
optogenetic arrays consist of both recording electrodes and 
fibers for optical stimulation. This design uses a novel light 
delivery system in which light pulses from a laser are 
directed onto a Digital Micromirror Device. These DMD 
chips consist of over 150,000 individually movable micro- 
meter size mirrors. Depending on the alignment of these 
mirrors, they will either allow the light to bypass the chip or 
reflect it into the components of a multicore fiberoptic, 
therefore allowing incredibly sophisticated patterns of 
activation. The multicore fibers consist of 30,000 individual 
strands. A number of these strands will be attached to 
movable electrode shafts allowing them to penetrate deep 
into the cortical columnar structure, thus negating the 
problem of light scatter by the tissue and allow very specific 
targeting of individual neurons. 

After the Chr-2 gene is delivered to the somatosensory 
and motor cortex of our monkeys through adeno-associated 
virus vector (AAV) injections performed using implanted 
cannulae, combined optical-stimulation/recording grids 
will be inserted in the cortex through the same cannulae. 
We expect that implanted animals will learn to use the 
spatiotemporal optogenetic stimulation delivered to their S1 
as BMI feedback. We will integrate this artificial sensory 
channel into our real-time BMI system, in which monkeys 
control the movements of primate virtual bodies, known as 
avatars, directly by brain activity while receiving sensory 
input through spatiotemporal optogenetic stimulation. 


BIMANUAL BMI 


So far, BMIs for arm movements involved only a single 
actuator that typically enacted reaching and grasping 
movements. Future BMIs will have to generate bimanual 
functionality. In their truest sense, bimanual movements are 
temporally and spatially coordinated movements of both 
upper limbs. At the DUCN, we are designing a BMI capable 
of extracting the motor commands to enact such coordina- 
tion. The starting point in developing a bimanual BMI is to 
understand the underlying cortical processes. Bimanual 
processes activate different subsets of neurons than unim- 
anual tasks.°*”’ As previous BMI research has focused on 
unimanual tasks, a key component of transitioning to 
successful bimanual control will be recognizing and adapt- 
ing to the differences in the cortical representation of this 
behavior. Unimanual movement initiation and coordination 
have often been shown to be correlated to neural activity in 
the contralateral hemisphere of the motor cortex. However, 
in tasks involving both arms, activated cortical areas are less 
clearly defined and have greater interhemispheric interac- 
tions. Interhemispheric connections of premotor regions 
have been the subject of several investigations, and several 
findings point to the fact that these regions exhibit much 
greater interhemispheric connectivity than M1.’ At least 
five cortical regions show activation related to inter-limb 
coordination: dorsal premotor cortex (PMd); cingulate 
motor area (CMA); supplementary motor area (SMA); 
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posterior parietal cortex (PPC); and M1.°° We will implant 


these areas to build the first bimanual BMI. 

In our bimanual BMI experiment, monkeys have to move 
two avatar hands within an initial target that appears in the 
center of the screen. When this task is learned, the monkey 
will learn a similar behavior, except that two peripheral 
targets will appear, and the monkeys will have to guide 
each avatar hand to the separate targets. We expect that 
monkeys will eventually learn to control such bimanual 
movements through a BMI, without engaging any overt 
movements of their limbs. 


BIPEDAL LOCOMOTION 


Previous BMI studies focused predominantly on the 
behavioral tasks in which an artificial actuator enacted 
upper extremity movements, such as reaching and grasping. 
Except for a few studies,*°' virtually no attempts have 
been made to translate BMI technology to tasks enacting 
motor functionality of lower extremities. Yet, deficits or the 
complete loss of the ability to walk present a considerable 
problem. Such deficits commonly result from spinal cord 
injury,°** neurologic diseases®”” and limb loss.°° Surveys 
of paraplegic patients showed that they prioritized walking 
and trunk stability among the most desired motor functions 
they would like to be restored.®’”° Quadriplegic patients 
prioritized arm and hand function.®’ Thus, developing 
neural prosthetic devices for restoration of leg mobility is as 
important as developing neuroprosthetics for resuming arm 
and hand movements, and for some categories of patients it 
is their top priority. Such neuroprosthetic devices would 
clearly have a major impact on the community of patients 
suffering from leg paralysis. 

In our treadmill locomotion setup, rhesus macaques walked 
bipedally on a custom modified treadmill.* We tracked 
movements of the right legs of the monkeys using a wireless, 
video-based tracking system developed in our laboratory.”! 
We then added a second tracking system to the setup and 
tracked movements of both legs. Leg kinematics and electro- 
myographies (EMGs) of leg muscles were reconstructed from 
neuronal ensemble activity in real time using the Wiener 
filter.'°!”?°" While the individual neuronal firing rates were 
highly variable from step to step at the millisecond scale, 
combining the activity of many neurons using a BMI decoder 
produced accurate predictions of leg movements. Extraction of 
neural information performed separately for different cortical 
areas showed that walking parameters could be predicted 
using neuronal activity recorded in either M1 or S1 contral- 
ateral to the right leg, as well as ipsilateral M1. In these 
experiments, neuronal ensemble activity was simultaneously 
recorded from all 384 channels. 

Using the locomotion setup, we also demonstrated real-time 
BMI control of bipedal walking in a robot.” We sent the 
predictions of monkey leg kinematics to the Advanced 
Telecommunications Research Institute (ATR), Kyoto, Japan, 
where our collaborators set their humanoid 51-degree of freedom 
robot (manufactured by SARCOS, Salt Lake City, UT, USA) to 
reproduce monkey locomotion patterns. The monkey received 
visual feedback of the robot’s movements on a video screen. 


POSTURE AND BALANCE 


BMI for restoration of lower limb function is fundamen- 
tally different from the BMI for upper limbs because it has 
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to not only replicate basic gait and stepping functions, but 
also adapt to postural control. In addition to developing a 
BMI for bipedal locomotion, we have developed a proof of 
concept BMI for postural control.” Using this experimental 
setup, in which monkeys stood on a platform that produced 
abrupt horizontal movements and evoked postural pertur- 
bations, we demonstrated recently that cortical neurons 
modulate their firing rate in relation to changes in posture. 
We then applied BMI decoders to extract information about 
these changes from cortical ensemble activity. The platform 
was driven either rhythmically, allowing the animal to 
anticipate the upcoming perturbation, or with a random 
time delay between movements so that the animal was 
unable to predict when the next movement would come. 
Similarly to our treadmill experiments, single unit analysis 
of the neural activity showed that a vast majority of cortical 
neurons (mostly in M1 and S1 representations of the legs) 
were highly modulated in association with compensatory 
postural reactions. These modulations were directionally 
tuned: neuronal rates changed differently depending on the 
direction of platform displacement. The BMI decoder based 
on a linear model predicted postural disturbances with 
good signal-to-noise ratios. Importantly, the decoder 
yielded different results depending on anticipated versus 
unanticipated platform displacements. Model parameters 
were different in these two cases, and anticipated displace- 
ments were predicted with higher accuracy than antici- 
pated. These results show the feasibility of a BMI for 
restoration of upright posture. The concrete implementation 
of such a BMI should be based on shared control rather than 
relying solely on cortical modulations for postural control. 


INTEGRATING FUNCTIONAL ELECTRICAL 
STIMULATION (FES) TO THE BMI PARADIGM 


The issue of directly actuating the movements of 
paralyzed legs has been addressed in many clinical studies 
in spinal cord injury patients. Robotic orthoses”? and 
functional electrical stimulation (FES) devices”*”” have been 
introduced as therapies for leg paralysis. FES devices for the 
lower extremities date back to the 1960s when the first FES 
application to restore standing was developed.*” In this 
study, paraplegic subjects were able to stand after FES was 
applied to their quadriceps and gluteus maximus muscles. 
Also in the 1960s, FES of the common peroneal nerve was 
used to correct foot drop during the swing phase of the 
gait.®' Foot drop stimulators have been commercialized.***° 
This technique is used in a Parastep™ system developed by 
Sigmedics, Inc, Fairborn, OH, USA. 185 At present, a variety 
of FES methods have been developed that allow stimulation 
of multiple muscle groups with surface or intramuscular 
electrodes. Importantly, FES systems have many positive 
therapeutic effects on patients with incomplete spinal cord 
injury (SCI).”” Unfortunately, current FES systems allow for 
only crude movement such as short-distance ambulation or 
short-term standing, require support from a walker or 
similar tool, and are primarily offered as a way to avoid 
medical complications of prolonged limb inactivity rather 
than as an alternative to locomotion restoration. Addi- 
tionally, such systems require manual actuation by the user, 
resulting in a limited and arguably non-intuitive way to 
restore leg function. 

Building on our success in BMIs that enact bipedal 
locomotion and postural control, we are currently developing 
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a cortically driven FES system for the restoration of walking. 
This novel BMI will be tested in a monkey model of bipedal 
walking. Our objective is to build a BMI-driven FES system 
that produces bipedal locomotion patterns by converting 
cortical ensemble activity into stimulation patterns that drive 
leg muscles. We suggest that BMI control over bipedal 
locomotion can be established by recording large-scale 
cortical activity from the sensorimotor cortex, extracting 
locomotion patterns from the raw cortical signals and 
converting them into trains of FES applied to multiple leg 
muscles. 


WHOLE-BODY NEUROPROSTHETIC 


As follows from our results on BMIs that enact leg 
movements, BMIs for the whole body are likely to become a 
real possibility in the near future. We propose the develop- 
ment of a whole-body BMI in which neuronal ensemble 
activity recorded from multiple cortical areas in rhesus 
monkeys controls the actuators that enact movements of 
both upper and lower extremities.*” This BMI will be first 
implemented in a virtual environment (monkey avatar) and 
then using a whole-body exoskeleton. In these experiments 
we will also examine the plasticity of neuronal ensemble 
properties caused by their involvement in the whole-body 
BMI control and the ability of cortical ensembles to adapt to 
represent novel external actuators. 

Furthermore, we will also explore the ability of an animal 
to navigate a virtual environment using both physical and 
neural control that engages both the upper and lower limbs. 
The first phase of these experiments will be to train the 
animals to walk in a bipedal manner on a treadmill while 
assisting the navigation with a hand steering control. We 
have already built a virtual environment needed for the 
monkey to navigate using 3D visualization software. Within 
this environment, the monkey’s body is represented by a 
life-like avatar. This representation is viewed in the third 
person by the monkey and employs real-world inverse 
kinematics to move, allowing the avatar’s limbs to move in 
close relation to the experimental animal. 

Initially, the direction that the avatar is facing will be 
dictated by the monkey moving a handlebar with its hands. 
As the animal moves the handlebar left or right, the avatar 
will rotate in the corresponding direction. The avatar’s legs 
will mimic the exact motion of the monkey’s legs on the 
treadmill. The simplest task will be for the animal to simply 
move the avatar forward to an object that represents a 
reward, a virtual fruit. Virtual fruits will appear at different 
angular positions relative to the monkey, which will let us 
measure the neuronal representation of navigation direction 
and modulations in cortical arm representation related to 
the steering. The monkey will have to make several steps 
while steering in the required direction to approach a 
virtual reward and to obtain an actual reward. The next set 
of experiments will allow the animal to control the virtual 
BMI in a manner similar to how we anticipate that the 
eventual application will be used: with no active movement 
of the subject’s body parts. The animals will use the neural 
control of the environment to obtain rewards when they are 
seated in a monkey chair. We expect that the monkey will be 
able to generate periodic neural modulations associated 
with individual steps of the avatar even though it does not 
perform any actual steps with its own legs. 
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Finally, we will use the algorithms developed in these 
experiments to control a full-body monkey exoskeleton in a 
non-human primate which has been subjected to a spinal 
cord anesthetic block to produce a temporary and reversible 
state of quadriplegia. This exoskeleton will encase the 
monkey’s arms and legs. It will be attached to the monkey 
using bracelets molded in the shape of the monkey’s limbs. 
A full body exoskeleton prototype will be utilized. The basic 
design and controller will be based on the humanoid robot, 
Computational Brain (CB).?”*° The exoskeleton will provide 
full-sensory feedback to the BMI set up—joints position/ 
velocity/torque, ground contacts and orientations. In BMI 
mode, the exoskeleton will guide the monkey’s limbs with 
smooth motions while at the same time monitoring its range 
of motions to ensure it is within the safety limits. This 
demonstration will provide the first prototype of a neural 
prosthetic device that would allow paralyzed people to 
walk again. 


CONCLUSIONS 


BMI technology offers a revolutionary treatment for 
paralysis. Recent studies suggest that BMIs have the 
potential to restore mobility to both upper and lower 
extremities and to enable a range of motor tasks, from arm 
reaching and grasping, to bipedal locomotion and balance. 
Moreover, it is feasible to enhance BMIs with an artificial 
somatosensory feedback either through ICMS or optoge- 
netic stimulation. We envision that multidisciplinary BMI 
research will lead to the creation of whole-body neural 
prosthetic devices aimed at restoring full, essential mobility 
functions to paralyzed patients. 
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Abstract: Brain-machine interfaces (BMIs) hold promise for the restoration of body mobility in 
patients suffering from devastating motor deficits caused by brain injury, neurological diseases, and 
limb loss. Considerable progress has been achieved in BMIs that enact arm movements, and initial 
work has been done on BMIs for lower limb and trunk control. These developments put Duke 
University Center for Neuroengineering in the position to develop the first BMI for whole-body 
control. This whole-body BMI will incorporate very large-scale brain recordings, advanced decoding 
algorithms, artificial sensory feedback based on electrical stimulation of somatosensory areas, virtual 
environment representations, and a whole-body exoskeleton. This system will be first tested in 
nonhuman primates and then transferred to clinical trials in humans. 


Keywords: brain—machine interface; brain-machine-brain interface; intracortical microstimulation; bidi- 
rectional brain—-machine interface; neuroprosthetic feedback; artificial sensation; active touch; locomo- 
tion; functional electrical stimulation; bimanual; multielectrode implant; primary motor cortex; primary 
somatosensory cortex; exoskeleton; posture; balance. 


Introduction 2001; Nicolelis and Lebedev, 2009; Schwartz 

et al., 2006; Wessberg et al., 2000) offer a transla- 
Brain—machine interfaces (BMIs) (Andersen tional solution to the problem of restoring mobil- 
et al., 2004; Birbaumer and Cohen, 2007; Fetz, ity to millions of people who suffer from paralysis 
2007; Lebedev and Nicolelis, 2006; Nicolelis, caused by neurological injuries, neurodegenera- 


tive diseases, or limb loss (Paddock, 2009). Only 
limited treatment options are available to these 
patients, and often their condition cannot be 
improved or ameliorated (Dobkin and Havton, 
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2004; Fouad and Pearson, 2004). BMIs hold 
promise to revolutionize the treatment of paraly- 
sis, as they strive to repair the damaged neural 
circuitry by bypassing the site of the lesion and 
establishing direct neural control of artificial tools 
by the activity of intact brain areas, such as the 
primary motor cortex (M1), which in many cases 
remain capable of generating motor commands 
despite being disconnected from the body 
effectors (Mattia et al., 2009). BMI research has 
expanded rapidly during the past decade 
(Lebedev and Nicolelis, 2006; Nicolelis and 
Lebedev, 2009), generating high expectations for 
potential clinical applications. Proof-of-concept 
BMIs have been tested in rodents (Chapin et al., 
1999), nonhuman primates (Carmena et al., 2003; 
Moritz et al., 2008; Taylor et al., 2002; Velliste 
et al., 2008; Wessberg et al., 2000), and human 
subjects (Allison et al., 2007; Birbaumer and 
Cohen, 2007; Hochberg et al., 2006; Kennedy 
and Bakay, 1998; Patil et al., 2004; Pfurtscheller 
and Neuper, 2006). BMI systems developed at the 
Duke University Center for Neuroengineering 
(DUCN) during the past 12 years have made it pos- 
sible to control many motor functions by neuronal 
ensemble activity recorded with chronic implants, 
ranging from arm reaching and grasping move- 
ments (Carmena et al., 2003; Lebedev et al., 2005; 
Wessberg et al., 2000) to bipedal locomotion 
(Cheng et al., 2007a; Fitzsimmons et al., 2009). 
Moreover, recently we have demonstrated for the 
first time brain—machine-brain interfaces (BMBIs) 
that incorporate somatosensory feedback loops 
that transmit information from the actuator to the 
brain (O’Doherty et al., 2009, 2010). These devel- 
opments have put us in the position to develop 
the first whole-body neuroprosthetic for severely 
paralyzed patients. 


Whole-body neuroprosthetic 
Previous BMI studies focused predominantly on 


behavioral tasks in which an artificial actuator 
enacted upper extremity movements, such as 


reaching and grasping. Except for a few studies 
(Fitzsimmons et al., 2009; Pfurtscheller et al., 
2006; Prilutsky et al., 2005), virtually no attempts 
have been made to translate BMI technology to 
tasks enacting motor functionality of lower 
extremities and the trunk. Yet, deficits or the 
complete loss of the ability to walk presents a 
considerable problem for millions of patients 
worldwide. Such motor deficits commonly result 
from spinal cord injury (Dietz, 2001; Rossignol 
et al., 2007; Scivoletto and Di Donna, 2009) and 
neurological diseases (Boonstra et al., 2008; 
Pearson et al., 2004). Surveys of paraplegic 
patients showed that they prioritized walking 
and trunk stability among desired mobility 
functions (Anderson, 2004; Brown-Triolo et al., 
2002). Quadriplegic patients prioritized arm and 
hand function (Anderson, 2004). Thus, develop- 
ing neural prosthetic devices for restoration of 
leg mobility is as important as developing neuro- 
prosthetics of the arm and hand, and for some 
categories of patients, it is one of their main 
priorities in terms of rehabilitation gain. In this 
context, a whole-body neuroprosthetic device 
would clearly have a major impact on the commu- 
nity of people suffering from many types of body 
paralysis. 

As it defines its strategic mission for the next 
decade, the DUCN has elected as its main 
priority to construct, test, and implement a clini- 
cal version of a whole-body BMI. We envision a 
whole-body BMI as a neuroprosthetic system in 
which neuronal ensemble activity sampled 
simultaneously from multiple cortical areas con- 
trols the actuators that generate movements 
of both upper and lower extremities (Winans 
et al., 2010). In theory, the actuators can be 
implemented as a whole-body exoskeleton or a 
set of devices for functional electrical stimulation 
(FES). As the main project, the DUCN has opted 
for the first solution, building a whole-body exo- 
skeleton in partnership with Dr. Gordon Cheng’s 
group at the Technical University of Munich. 

In our efforts to implement the first whole- 
body neuroprosthetic device for clinical use, the 
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first step will be to test a monkey version of such 
a BMI. This prototype will take into account all 
findings and technological developments obtained 
by our laboratory during the past decade. We 
expect that this whole-body BMI will provide 
animals with the ability to navigate a virtual envi- 
ronment through the control of a realistic repre- 
sentation of a monkey body. All the movements 
of this life-like computational avatar will be 
enacted by the animal’s brain activity. Indeed, in 
this experimental paradigm, monkeys will learn to 
control the avatar’s body movements under BMI 
control and to perform a series of tasks that require 
both upper and lower limb coordinated move- 
ments, including reaching and grasping virtual 
objects, selecting objects with different texture, 
bimanual object manipulation, and autonomous 
bipedal locomotion. Once monkeys achieve a high 
degree of proficiency in interacting with such a 
whole-body BMI in a virtual environment, the 
same technology will be then transferred to control 
areal whole-body exoskeleton that the animals will 
wear. These latter primate experiments, in which 
animals learn to control an exoskeleton through a 
BMI, will be developed in parallel with the creation 
of a virtual environment for patients that will use 
magnetoencephalography to generate the type of 
brain-derived signals needed for these patients to 
learn how to operate a realistic avatar of their 
own body (see Fig. 1). Through this sequence of 
animal and clinical experiments, we intend to gen- 
erate a prototype neural prosthetic device for 
whole-body control which will be then translated 
to clinical trials and, eventually, enable severely 
paralyzed people to walk again by 2016. 

Below, we review recent results obtained at the 
DUCN that support our contention that a whole- 
body BMI can be implemented by this deadline. 


BMI components 
The basic components of a BMI system are 


exemplified by the now classical paradigm that 
enacts direct control of robotic arm reaching 
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movements based on the combined cortical activ- 
ity of hundreds of cortical neurons (Carmena 
et al., 2003; Lebedev and Nicolelis, 2006; Lebedev 
et al, 2005; Nicolelis and Lebedev, 2009; 
Wessberg et al., 2000). In this BMI paradigm, 
the electrical activity of large populations of 
motor cortical neurons is recorded by chronically 
implanted multielectrode arrays and converted 
into control signals that drive a robotic arm. 
Concurrently, visual and/or somatosensory feed- 
back signals from the robot are delivered back 
to the brain as either natural visual stimuli or, 
as in our recent demonstrations, intracortical 
microstimulation (ICMS) of the primary somato- 
sensory cortex (Sl) (Lebedev and Nicolelis, 
2006; O’Doherty et al., 2009, 2010). The pro- 
cessing of neuronal ensemble activity is per- 
formed by a series of real-time BMI decoding 
algorithms which translate a myriad of neuronal 
spikes into the kinematic and dynamic parameters 
required to generate the robotic arm’s move- 
ments, according to the voluntary motor intent- 
ions of the subject. The BMI setup also includes 
the data acquisition system, the computer cluster 
running multiple decoding models in real time, 
the robot arm, the visual display, and a sensory 
feedback loop from the actuator to the brain. 


Large-scale neuronal recordings 


The major prerequisite for the performance of a 
neuroprosthetic device to be versatile, accurate, 
and stable, and to allow simultaneous motor con- 
trol of both lower and upper extremities, is that 
multiple brain areas should be implanted and 
large-scale neuronal activity sampled from those 
areas simultaneously during operation of a 
whole-body BMI (Nicolelis and Lebedev, 2009; 
Nicolelis et al., 2003). During the past two 
decades, advanced electrophysiological methods 
have allowed recording from progressively larger 
samples of single neurons in behaving animals 
(Churchland et al., 2007; Nicolelis and Ribeiro, 
2002; Nicolelis et al., 1995, 2003). With the 
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Fig. 1. Schematic illustration of a human whole-body neuroprosthetic based on magnetoencephalographic recordings and functional 
magnetic resonance imaging that enables a paralyzed human to operate a realistic avatar of their own body in a virtual environment. 
Brain signals are processed by a powerful computer cluster and converted into commands that drive a life-like avatar. This 
technology will be used to develop and test a brain-controlled whole-body navigation system and will be eventually translated 


into a mind-operated whole-body exoskeleton. 


present generation of planar multielectrode arrays, 
which can be chronically implanted in the brain, we 
can record the extracellular electrical activity of 
several hundreds of cortical cells simultaneously 
in a behaving rhesus monkey ( 


). Due to a rapid progress 
in the methods for large-scale recordings ( 
), the number of simultaneously sampled 


neural channels is expected to rise to several 
thousands in the next decade. Indeed, recently, 
we have started to employ a new type of device, 
called the three-dimensional (3D) recording cube, 
which provides recording sites throughout the ver- 
tical shaft of each microwire bundle. Preliminary 
results suggest that such a 3D recording cube could 
allow neuronal yield to reach several tens of 
thousands of neurons in the near future. Given that 
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our previous studies have clearly demonstrated 
that the accuracy of extraction of motor intentions 
from the brain improves with the number of 
neurons recorded (Carmena et al., 2003; Lebedev 
et al., 2007; Nicolelis and Lebedev, 2009; Santucci 
et al., 2005; Wessberg et al., 2000), and given that 
the number of required neurons increases as more 
motor parameters are simultaneously extracted 
(Fitzsimmons et al., 2009), the advent of the 3D 
recording cube, and other technologies like it, could 
prove vital for the translation of experimental 
discoveries into clinical practice. Incidentally, the 
central experimental finding obtained through 
BMI research, the confirmation that distributed 
ensembles of neurons define the true physiological 
unit of the mammalian central nervous system 
(Nicolelis and Lebedev, 2009), will also play a major 
role in the road that will take us to clinical fruition. 

A brief description may help clarify the 
imposing need for increasing the size of cortical 
recording samples in order to obtain the level of 
motor control needed to operate a whole-body 
exoskeleton. Currently, our most sophisticated 
BMIs provide a peak signal-to-noise ratio (SNR) 
of 10dB using approximately 100 neurons 
(Fitzsimmons et al., 2009; Li et al., 2009). This 
corresponds to 5 bits/s bandwidth. In our estima- 
tion, SNRs of about 20-30 dB are needed for a 
neuroprosthetic device to generate useful move- 
ments (see Fig. 2). Assuming that noise decreases 
as the inverse of the square root of the number of 
neurons, a tenfold increase in the number of 
simultaneously recorded neurons is needed to 
achieve an improvement of 10 dB. A control sig- 
nal with 20-dB fidelity will then require 1000 
neurons recorded simultaneously. For achieving 
our long-term goal of reaching an SNR of 40 dB, 
and control a whole-body exoskeleton, recording 
samples between 50,000-100,000 cortical neurons 
will be needed. Further, to control the move- 
ments of multiple prosthetic limbs, each of which 
requires enacting of many degrees-of-freedom 
(DOF), a whole-body neuroprosthetic will call 
for even higher neuronal samples. Thus, demands 
for large samples of neurons, located in multiple 
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cortical areas, will remain a central bottleneck 
for the development of clinical neuroprosthetics 
for the foreseeable future. 

Another key issue to be dealt with is the ques- 
tion of the longevity of chronic multielectrode 
recordings. Heretofore, our state-of-the-art multi- 
electrode implants reliably sample large-scale 
extracellular cortical electrical activity for at least 
2 years in rhesus monkeys (Fitzsimmons et al., 
2009) and at least 6.5 years in New World 
monkeys, like the owl monkey (Sandler et al., 
2005). We have used a variety of designs for such 
multielectrode probes. Typically, electrode pene- 
tration sites are spaced with a 1-mm separation. 
The electrodes can be fixed or movable. In our 
recent design, we arranged stainless steel 
microwires into subsets (pairs or triplets) sitting 
inside guiding tubes (Winans et al., 2010). Each 
subset can be moved independently driven by a 
microscrew. Given that the electrodes in each 
subset have different lengths, the array samples 
from a 3D volume; that is, how we started to test 
what became known as the 3D recording cube. 
Currently, we are using this new cube design as 
a base for the development of the next-generation 
implants that will increase the number of record- 
ing channels to about 1500 per cube. That is 
because a 10 x 10 array will gain up to 15 record- 
ing contacts per vertical component, using micro- 
wire assemblies of different lengths. Further, to 
minimize the displacement of the nervous tissue 
and to minimize microglial and other immune 
responses to the foreign material during the surgi- 
cal implantation of these recording cubes, we will 
be removing the structural elements after implan- 
tation, leaving only small diameter microwires 
within the neuronal tissue. Small-diameter 
microwires will be guided into the brain with a 
strong central shaft which will be removed later. 
To distinguish the type of cortical activity obtained 
with this new technology, we have coined the term 
very large scale brain activity (VLSBA), meaning 
extracellular neuronal activity generated by 
populations of more than 50,000 single neurons 
recorded simultaneously. By developing the means 
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(C= 10 for uncorrelated neurons, C,=3.7 for our data) 


Fig. 2. An estimation of the dependency of the BMI signal-to-noise ratio (SNR) on the neuronal ensemble size. The scatter plot on 
the left shows a linear dependency of the SNR (in dB) on the logarithm of the number of neurons recorded. The data from 
Fitzsimmons et al. (2009) was used to construct this plot. This dependency is in correspondence with the equation shown at the 
bottom. The table on the right shows the estimates of the number of neurons needed to achieve certain performance benchmarks. 


to record from VLSBA, we expect to generate the 
type of motor control signals that will be capable 
of driving a whole-body BMI. 


BMI decoders 


The success of any BMI system depends to a sig- 
nificant degree on the choice of BMI decoders 
that extract motor parameters from the sample 
of neuronal electrical activity recorded in real 
time. In our current studies, online processing of 
large-scale brain activity is achieved through an 
integrated BMI suite that incorporates the 
recording and stimulation hardware, as well as a 
computer cluster employed for all real-time pro- 
cessing of the massive stream of neurophysiologi- 
cal data generated in each of these experimental 
sections (Fitzsimmons et al., 2009; O’Doherty 
et al., 2009). This BMI suite can simultaneously 
run several neuronal decoders, including the 
unscented Kalman filter (Li et al., 2009), the Wie- 
ner filter, multiple artificial neural networks, and 
discrete state Bayesian algorithms (Li et al., 


2010). Each motor parameter can be extracted 
by one of these filters or can represent a mixture 
of the outputs of individual decoders. During the 
extraction or arm reaching movements, the 
unscented Kalman filter outperforms the other 
decoding algorithms (Li et al., 2010). This 
decoder is based on nonlinear models of neural 
tuning and prior knowledge about movement 
patterns. It is also enhanced by a short history of 
the arm movements. We have also developed a 
multiple model-switching paradigm (Fitzsimmons 
et al., 2009) that uses different submodels to 
extract motor variables in particular behavioral 
states (e.g., reaching vs. grasping or walking for- 
ward vs. walking backward). The simplest 
switching model consists of three linear decoding 
models: a model for predicting state 1 (e.g., walk- 
ing forward), a model for state 2 (e.g., walking 
backward), and the paradigm predictor model 
(the switch). When the paradigm predictor 
determines a particular state, an appropriate sub- 
model is used to produce the output, and when a 
different state is detected, the other submodel is 
used. 
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In the most straightforward implementation of 
BMI decoding, the kinematics of several limb 
joints is extracted from the combined neuronal 
electrical activity and converted into movements 
of the particular mechanical, electronic or virtual 
actuator employed in the experiment at hand. 
This general approach, however, is not optimal 
for clinical applications of BMIs because the user 
is required to control each DOF continuously and 
independently. In a more practical solution, the 
control over an external actuator is shared 
between the subject brain’s activity and robotic 
controls (Kim et al., 2005). During this shared 
control, the subject’s brain is in charge primarily 
of high-order control of movements (when to ini- 
tiate movement and where to move), whereas the 
low-level coordination of the movement is per- 
formed by an autonomous controller. 

Recently, we have developed a BMI decoder 
that autonomously adjusts its performance during 
long-term recordings and compensates for non- 
stationarities in neuronal inputs (Li et al., 2010). 
This decoder uses a Bayesian regression self- 
training method for updating the parameters for 
an unscented Kalman filter. To allow updates on 
subsets of neurons and to allow addition of newly 
discovered neurons, we approximated the proba- 
bility distribution on the tuning model parameters 
using a factorized distribution and computed 
the Bayesian regression solution. We tested the 
performance of this filter in rhesus monkeys that 
learned to perform reaching movements under 
the BMI control. Over the course of 29 days, 
Bayesian regression  self-training maintained 
control accuracy better than decoding without 
updates. This BMI decoder will be used in the 
whole-body BMI to provide long-term stability. 


Bimanual control 


A whole-body neuroprosthetic will have to 
enable independent control of two prosthetic 
arms. Such BMI control has not been achieved 
before. The majority of BMIs developed so far 
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involved only a single actuator (computer cursor 
or a robotic arm) that enacted arm reaching 
movements under the control of the subject’s 
brain activity. We are currently exploring the 
use of BMIs for bimanual actuator control. As 
in other applications, the starting point in devel- 
oping a bimanual BMI is to understand the 
neurophysiological processes that underlie biman- 
ual behaviors in multiple cortical areas. Bimanual 
operations engage neuronal populations which 
are different from those engaged by unimanual 
movements (Kazennikov et al., 1999; Nakajima 
et al., 2007; Obhi and Goodale, 2005; Rokni 
et al., 2003; Rouiller et al., 1994). When both 
arms are involved, motor areas are activated in 
both hemispheres, and greater interhemispheric 
interactions occur. Five cortical areas involved in 
interlimb coordination are of particular interest 
for bimanual BMIs: dorsal premotor cortex 
(PMd), cingulate motor area, supplementary 
motor area, posterior parietal cortex, and M1 
(Kermadi et al., 2000). 

We have conducted preliminary experiments to 
investigate if rhesus monkeys could learn bimanual 
control of a life-like avatar. In these experiments, 
one monkey learned to manipulate two joysticks 
to move two avatar hands toward virtual objects. 
This learning occurred rapidly, suggesting that 
the monkey readily associated itself with the 
avatar. This observation strengthens our expecta- 
tion that monkeys will be able to learn to 
control such bimanual movements through a 
BMI, without engaging any overt movements of 
their limbs. 


Bipedal locomotion 


Our laboratory pioneered BMIs that reproduce 
kinematics of leg movements during bipedal loco- 
motion (Fitzsimmons et al., 2009). Previously, 
both BMI research and neurophysiological stud- 
ies in awake, behaving monkeys focused predom- 
inantly on the behavioral tasks that involved arm 
movements and arm representation in the brain. 
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Neurophysiology of lower extremity control has 
been virtually neglected in nonhuman primates. 
Yet, a complete loss of the ability to walk is com- 
monly caused by spinal cord injury (Dietz, 2001; 
Rossignol et al., 2007; Scivoletto and Di Donna, 
2009), neurological diseases (Boonstra et al., 
2008; Pearson et al., 2004), and limb loss (Pasquina 
et al., 2006). Developing a neuroprosthetic for 
lower limb control will clearly be very important 
for the treatment of whole-body paralysis. 

In our initial study on the BMI for bipedal loco- 
motion, rhesus monkeys walked bipedally on a 
treadmill (Fitzsimmons et al., 2009). We tracked 
leg movements using a video-based tracking system 
developed in our laboratory (Peikon et al., 2009). 
Leg kinematics and EMGs of leg muscles were 
extracted from linearly combined cortical ensemble 
activity in real time using a modified Wiener filter 
algorithm (Carmena et al., 2003; Haykin, 2002; 
Lebedev et al., 2005; Wessberg et al., 2000). Record- 
ing implants were placed in the leg representation 
area of M1. We found that the individual neuronal 
firing rates were highly variable from step to step. 
However, after the activity of many (several 
hundreds) neurons was combined using 100-ms 
bins, accurate extractions of the kinematics of leg 
movements (X,Y coordinates of the joints, joint 
angles) were produced. Walking parameters could 
be extracted using neuronal activity recorded in 
either M1 or S1 contralateral to the right leg, as well 
as ipsilateral M1. 

Simultaneous extraction of many parameters of 
bipedal locomotion essentially depended on the 
size of the neuronal population. In this analysis, 
random neuronal subpopulations were pooled 
from the entire recorded population and used to 
predict several locomotion parameters simulta- 
neously. The accuracy of simultaneous pre- 
dictions was characterized by the normalized 
accuracy for the least well-predicted parameter. 
We found that smaller neuronal populations 
could predict only a few parameters simulta- 
neously, and larger populations were required 
for predicting many parameters. Moreover, larger 


neuronal populations were required to predict 
complex patterns of walking compared to more 
simple walking patterns. For example, the num- 
ber of neurons required to achieve 95% of maxi- 
mum prediction accuracy for the X position of 
the ankle clearly increased when walking 
conditions of higher complexity were required to 
complete a task (intermittent walking forward 
and backward, walking at different speeds). 
On average, 60 neurons were sufficient for 
predicting constant-speed walking in the forward 
direction. However, 90 neurons were needed to 
achieve this level of accuracy for variable-speed, 
forward walking, 95 neurons were required for 
extracting backward walking at constant speed, 
115 neurons were required for predicting back- 
ward walking at variable speed, and extracting 
variable-speed bidirectional walking required 
110 neurons. 

Using our BMI for extracting patterns of 
bipedal locomotion, we demonstrated BMI con- 
trol of bipedal walking in a robot (Cheng et al., 
2007a) in real time. To achieve this goal, real- 
time predictions of monkey leg kinematics, 
derived from combined cortical activity, were 
transmitted through a dedicated Internet connec- 
tion to ATR laboratories, in Kyoto, Japan, where 
our collaborators led by Cheng and Kawato set 
their humanoid 51-DOF robot (CB-1, man- 
ufactured by SARCOS) to reproduce monkey 
locomotion patterns. Back at our laboratory at 
Duke University, the monkey received continu- 
ous visual feedback of the robot’s movements 
on a video screen placed in front of the animal. 
As we had observed in the case of BMI control 
of upper-limb extremities, after the treadmill 
was stopped, the monkey continued to use its 
brain activity to sustain bipedal walking in the 
robot without moving its own biological legs. This 
result supports our suggestion that a paralyzed 
person will be able to control a device for loco- 
motion (e.g., an exoskeleton) with his/her cortical 
activity alone, without the need to produce overt 
body movements. 
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Posture and balance 


Whole-body neuroprosthetics will not only have 
to produce stereotypical stepping but also adapt 
to postural control. As an advancement toward 
this goal, we have developed a proof-of-concept 
BMI for postural control (Tate et al., 2009). In 
these experiments, monkeys first learned to main- 
tain an upright posture on a platform. Then, the 
platform moved abruptly, generating a postural 
perturbation. The platform was driven either 
periodically, allowing the animal to anticipate 
the upcoming perturbation, or with a random 
time delay between movements so that the animal 
could not anticipate when the next displacement 
would occur. By analyzing samples of simulta- 
neously recorded cortical neurons, we observed 
that cells recorded in the leg representation of 
M1 and S1 clearly modulated their firing rate in 
response to platform displacement. These mod- 
ulations were directionally tuned. Linear BMI 
decoders were applied to extract the kinematics 
of platform movements from cortical ensemble 
activity. We found that the decoder performance 
was different, depending on whether the platform 
movements were anticipated or unanticipated. 
Anticipated displacements were extracted with 
higher accuracy than unanticipated. These results 
suggest that cortical control over posture and bal- 
ance can be added to a whole-body BMI. Optimal 
operation of such control probably can be aided 
by a shared control scheme (Kim et al., 2005). 


Functional electrical stimulation 


FES that activates the subject’s own muscles may 
be implemented in future whole-body neuro- 
prosthetics. FES devices have been already 
introduced to clinical practice as therapies for leg 
paralysis (Barbeau et al., 2002; Dobkin, 2007; 
Peckham et al., 1988; Thrasher and Popovic, 
2008) along with robotic orthoses (Colombo 
et al., 2000; Dollar and Herr, 2008; Ferris et al., 
2007). The first publications on a FES device for 
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helping to achieve an upright posture date back 
to the 1960s (Kantrowitz, 1960). Also in the 
1960s, FES of the common peroneal nerve was 
introduced to correct foot drop during the swing 
phase of the gate (Liberson et al., 1961). At pres- 
ent, a variety of FES methods are available for 
stimulation of multiple muscle groups using super- 
ficial or intramuscular electrodes. In particular, 
FES systems have been shown to have positive 
therapeutic effects in patients with incomplete 
spinal cord injury (Thrasher and Popovic, 2008). 

We suggest that FES technology can be 
incorporated in BMIs for the restoration of walk- 
ing and balance. In support of this suggestion, 
we have already obtained reliable predictions of 
the EMGs of leg muscles during locomotion 
(Fitzsimmons et al., 2009) from the combined 
electrical activity of populations of M1 and S1 
cortical neurons. Our next objective is to build a 
BMI-driven FES system that produces bipedal 
locomotion patterns by converting cortical ensem- 
ble activity into stimulation patterns that drive leg 
muscles. This approach can be used alone or in 
combination with robotic orthoses, such as 
exoskeletons. 


Sensorized neuroprosthetic 


Recently, we have reported our findings on the first 
BMBI. Such a paradigm expands on traditional 
BMIs by adding an artificial somatosensory feed- 
back channel that can deliver artificially created 
tactile signals, generated by either real sensors 
placed in a robotic hand or virtual ones added to 
an avatar arm, directly to the somatosensory cor- 
tex, via ICMS. In our first study on this subject 
(Fitzsimmons et al., 2007), we investigated whether 
multichannel ICMS of S1 could be discriminated 
by owl monkeys and whether ICMS is suitable for 
long-term usage. Owl monkeys were implanted 
with multielectrode arrays in several cortical areas, 
whereas S1 implants were employed to deliver spa- 
tiotemporal patterns of ICMS. The behavioral task 
progressed from a simple requirement of detecting 
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the presence of ICMS to the goal of discriminating 
spatiotemporal patterns created using four elec- 
trode pairs. We found that owl monkeys could 
learn to discriminate spatiotemporal patterns of 
ICMS of increasing complexity and guide their 
arm reaching movements based on this discrimina- 
tion. Moreover, spatiotemporal ICMS was efficient 
for many months. Interestingly, monkeys got pro- 
gressively better in learning novel microstimulation 
patterns. This result suggests that ICMS was 
incorporated in their brains as a new sensory channel 
and that they could generalize the general rule 
applied to communicate directly with their brains. 
In a series of studies conducted in rhesus 
monkeys, we used ICMS to guide monkeys’ 
behavior when they performed BMI-reaching 
tasks (O’Doherty et al., 2009, 2010). We followed 
our usual paradigm of training a BMI decoder 
during a manually performed task and then 
switched to brain control (Carmena et al., 2003; 
Lebedev et al., 2005). The tasks consisted of 
acquiring visual targets with a computer cursor 
or a life-like avatar hand. We recorded simulta- 
neously the electrical activity of 50-200 neurons 
in M1 and PMd. The monkeys learned to perform 
in BMI control with and without using the joy- 
stick. The innovative feature of these experiments 
was that ICMS of the S1 cortex was added 
to the BMI as an artificial sensory feedback. In 
the first study that implemented this approach 
(O’Doherty et al., 2009), ICMS served as a direc- 
tional cue. It informed the monkeys to which 
direction they had to move. To avoid an interfer- 
ence of ICMS electrical artifacts with BMI 
extractions, we segregated the stimulation and 
recording epochs. Monkeys operated such a 
BMBI with approximately the same accuracy as 
reported previously for simpler BMI designs. 
Moreover, in our recent study (O’Doherty 
et al., 2010), ICMS served as an artificial sense 
of active touch, as it conveyed to the monkeys 
the properties of virtual objects that the actuator 
(computer cursor or a virtual image of a monkey 
arm) touched. Monkeys controlled an avatar 
arm with a BMBI that derived motor commands 


from the M1 activity. ICMS patterns were deliv- 
ered to S1 each time the avatar touched virtual 
objects. Monkeys learned to search through sets 
of visually identical objects and select those with 
particular textures. These results suggest that 
future clinical neuroprosthetics can implement 
ICMS feedback to generate somatic perceptions 
from prosthetic limbs. 

In our future work, we will incorporate multi- 
ple channel ICMS feedback in the BMBIs for 
locomotion and whole-body control. We will also 
work on the incorporation of an artificial sense of 
position. Position sense is very important for clin- 
ical applications because ideal prosthetic limbs 
should feel as if they are natural extensions of 
the users’ bodies. Normally, positional signals 
are provided by muscles, joints, and skin 
afferents. This information ascends to the sensory 
areas of the brain where it is processed using dif- 
ferent coordinate frames, such as body- and exter- 
nal space-centered coordinates (Maravita et al., 
2003; Matthews, 1988). Given this complexity of 
cortical processing of proprioceptive information 
(Longo et al., 2010), it would be difficult for an 
artificial position sense to achieve such precise 
mapping from the arm joints to the brain somato- 
sensory map. Additionally, certain centrally gen- 
erated components of normal position sense, 
such as corollary discharges (Crapse and Sommer, 
2008), would be difficult to incorporate in such 
an implementation. Because of these foreseen 
difficulties in the straightforward implementation 
of an artificial position sense, we chose a simpler 
approach in which stimulation of S1 is not initially 
coupled to the orientation of the limb position but 
instead represents 3D spatial locations to which 
the subject is required to reach. In this approach, 
the subject starts with learning how ICMS of cor- 
tical somatotopic representations of the body is 
mapped to a 3D space. Such experimental design 
bears similarity to the studies on sensory substitu- 
tion in which visual information was conveyed by 
the stimulation of skin surfaces (Bach-y-Rita, 
1983; Bach-y-Rita and Kercel, 2003; Bach-y-Rita 
et al., 1969). We expect that similar to sensory 
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substitution using peripheral stimulation, training 
with stimulation of the somatosensory areas of 
the brain will eventually give rise to an artificial 
position sense that represents external space. 

In addition to ICMS as the method to sensorize 
a whole-body BMI, optogenetic methods will 
be probably, extensively used in the future. 
Optogenetics is based on genetically modified ion 
channels that respond directly to light (Zhang et al., 
2007). These light-gated ion channels, such as 
Channelrhodopsin-2 (Chr-2), allow precise, millisec- 
ond control of specific neurons (Boyden et al., 2005; 
Zhang et al., 2006). This technique eliminates most 
of the key problems associated with ICMS: there is 
no associated electrical artifact to interfere with the 
electrophysiological recordings, nor any tissue dam- 
age from the current injection. It also allows for finer 
control of the spatial pattern of activation. BMBIs 
that incorporate optogenetic stimulators would be 
superior to current designs in both the specificity 
and the long-term performance of the sensory loop 
and the quality of neuronal recordings. 


A whole-body exoskeleton 


In our research program, the definitive demonstra- 
tion of a whole-body neuroprosthetic will involve a 
subject that is able to use his/her VLSBA to control 
movements of an exoskeleton that encases the 
entire body. Currently, we have all components 
needed for this demonstration. We have demon- 
strated BMIs for arm reaching and leg locomotion 
in separate experiments. We have also implanted 
leg and arm representations of the sensorimotor 
cortex in both hemispheres (Winans et al., 2010). 
The exoskeleton technology required for this dem- 
onstration already exists, and a prototype for our 
initial testing in monkeys is close to be completed. 
In our primate studies, such exoskeleton will be 
attached to the monkey using bracelets molded in 
the shape of the monkey’s limbs. The basic design 
and controller will be based on the humanoid 
robot, CB-1 (Cheng et al., 2007a,b). This exoskele- 
ton will provide a rich sensory feedback stream to 
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the BMI setup, including measurements of joints 
position and velocity, as well as torque, ground 
contacts, and orientations. In the BMI mode, the 
exoskeleton will guide the monkey’s limbs with 
smooth motions, while monitoring its range of 
motions to ensure it is within the safety limits. This 
upcoming demonstration will provide the first proto- 
type of a neural prosthetic device that would allow, 
one day in the near future, paralyzed people to 
recover a wide range of desired mobility functions. 
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WHAT WE CAN DO 


NEUROENGINEERING 


Mind in 
Motion 


The idea that paralyzed people 
might one day control their limbs 
just by thinking is no longer 

a Hollywood-style fantasy 
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By Miguel A. L. Nicolelis 


IN 2014 BILLIONS OF VIEWERS WORLDWIDE MAY 
remember the opening game of the 
World Cup in Brazil for more than just 
the goals scored by the Brazilian national 
team and the red cards given to its adver- 
sary. On that day my laboratory at Duke 
University, which specializes in develop- 
ing technologies that allow electrical sig- 
nals from the brain to control robotic 
limbs, plans to mark a milestone in over- 
coming paralysis. 

If we succeed in meeting still formi- 
dable challenges, the first ceremonial 
kick of the World Cup game may be made 
by a paralyzed teenager, who, flanked by 
the two contending soccer teams, will 
saunter onto the pitch clad in a robotic 
body suit. This suit—or exoskeleton, as 
we call it—will envelop the teenager’s 
legs. His or her first steps onto the field 
will be controlled by motor signals origi- 
nating in the kicker’s brain and transmit- 
ted wirelessly to a computer unit the size 
of a laptop in a backpack carried by our 
patient. This computer will be responsi- 


ble for translating electrical brain signals 
into digital motor commands so that the 
exoskeleton can first stabilize the kick- 
er’s body weight and then induce the ro- 
botic legs to begin the back-and-forth co- 
ordinated movements of a walk over the 
manicured grass. Then, on approaching 
the ball, the kicker will visualize placing 
a foot in contact with it. Three hundred 
milliseconds later brain signals will in- 
struct the exoskeleton’s robotic foot to 
hook under the leather sphere, Brazilian 
style, and boot it aloft. 

This scientific demonstration of a radi- 
cally new technology, undertaken with 
collaborators in Europe and Brazil, will 
convey to a global audience of billions that 
brain control of machines has moved from 
lab demos and futuristic speculation to a 
new era in which tools capable of bringing 
mobility to patients incapacitated by inju- 
ry or disease may become a reality. We are 
on our way, perhaps by the next decade, to 
technology that links the brain with me- 
chanical, electronic or virtual machines. 
This development will restore mobility, 
not only to accident and war victims but 
also to patients with ALS (also known as 
Lou Gehrig’s disease), Parkinson’s and oth- 
er disorders that disrupt motor behaviors 
that impede arm reaching, hand grasping, 
locomotion and speech production. Neu- 
roprosthetic devices—or brain-machine 
interfaces—will also allow scientists to do 
much more than help the disabled. They 
will make it possible to explore the world 
in revolutionary ways by providing 
healthy human beings with the ability to 
augment their sensory and motor skills. 

In this futuristic scenario, voluntary 
electrical brain waves, the biological al- 
phabet that underlies human thinking, 
will maneuver large and small robots re- 
motely, control airships from afar, and 
perhaps even allow the sharing of 
thoughts and sensations of one individu- 
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IN BRIEF 


Brain waves can now control the 
functioning of computer cursors, 
robotic arms and, soon, an entire suit: 
an exoskeleton that will allow 

a paraplegic to walk and maybe 

even move gracefully. 

Sending signals from the brain’s 
outer rindlike cortex to initiate 
movement in the exoskeleton 
represents the state of the art 

for a number of bioelectrical 
technologies perfected 

in recent years. 

The 2014 World Cup in Brazil will 
serve as a proving ground for a 
brain-controlled exoskeleton if, as 
expected, a handicapped teenager 
delivers the ceremonial opening kick. 
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al with another over what will become a 
collective brain-based network. 


THOUGHT MACHINES 

THE LIGHTWEIGHT body suit intended for 
the kicker, who has not yet been selected, 
is still under development. A prototype, 
though, is now under construction at the 
lab of my great friend and collaborator 
Gordon Cheng of the Technical Universi- 
ty of Munich—one of the founding mem- 
bers of the Walk Again Project, a nonprof- 
it, international collaboration among the 
Duke University Center for Neuroengi- 
neering, the Technical University of Mu- 
nich, the Swiss Federal Institute of Tech- 
nology in Lausanne, and the Edmond 
and Lily Safra International Institute of 
Neuroscience of Natal in Brazil. A few 
new members, including major research 
institutes and universities all over the 
world, will join this international team in 
the next few months. 

The project builds on nearly two de- 
cades of pioneering work on brain-ma- 
chine interfaces at Duke—research that 
itself grew out of studies dating back to 
the 1960s, when scientists first attempted 
to tap into animal brains to see if a neural 
signal could be fed into a computer and 
thereby prompt a command to initiate 
motion in a mechanical device. Back in 
1990 and throughout the first decade of 
this century, my Duke colleagues and I pi- 
oneered a method through which the 
brains of both rats and monkeys could be 
implanted with hundreds of hair-thin 
and flexible sensors, known as micro- 
wires. Over the past two decades we have 
shown that, once implanted, the flexible 
electrical prongs can detect minute elec- 
trical signals, or action potentials, gener- 
ated by hundreds of individual neurons 
distributed throughout the animals’ fron- 
tal and parietal cortices—the regions that 
define a vast brain circuit responsible for 
the generation of voluntary movements. 

This interface has for a full decade 
used brain-derived signals to generate 
movements of robotic arms, hands and 
legs in animal experiments. A critical 
breakthrough occurred last year when two 
monkeys in our lab learned to exert neural 
control over the movements of a comput- 
er-generated avatar arm that touched ob- 
jects in a virtual world but also provided 
an “artificial tactile’ feedback signal di- 
rectly to each monkey’s brain. The soft- 
ware allowed us to train the animals to 


feel what it was like to touch an object 
with virtual fingers controlled directly by 
their brain. 

The Walk Again consortium—assisted 
by its international team of neuroscien- 
tists, roboticists, computer scientists, 
neurosurgeons and rehabilitation profes- 
sionals—has begun to take advantage of 
these animal research findings to create a 
completely new way to train and rehabili- 
tate severely paralyzed patients in how to 
use brain-machine interface technologies 
to regain full-body mobility. Indeed, the 
first baby steps for our future ceremonial 
kicker will happen inside an advanced 
virtual-reality chamber known as a Cave 
Automatic Virtual Environment, a room 
with screens projected on every wall, in- 
cluding the floor and ceiling. After don- 
ning 3-D goggles and a headpiece that 
will noninvasively detect brain waves 
(through techniques known as electroen- 
cephalography—EEG—and magnetoen- 
cephalography), our candidate kicker— 
by necessity a lightweight teenager for 
this first iteration of the technology—will 
become immersed in a virtual environ- 
ment that stretches out in all directions. 
There the youngster will learn to control 
the movements of a software body avatar 
through thought alone. Little by little, the 
motions induced in the avatar will in- 
crease in complexity and will ultimately 
end with fine-motor movements such as 
walking on a changing terrain or un- 
screwing a virtual jelly jar top. 


PLUGGING INTO NEURONS 
THE MECHANICAL movements of an exo- 
skeleton cannot be manipulated as readi- 
ly as those of a software avatar, so the 
technology and the training will be more 
complicated. It will be necessary to im- 
plant electrodes directly in the brain to 
manipulate the robotic limbs. We will 
need not only to place the electrodes un- 
der the skull in the brain but also to in- 
crease the number of neurons to be 
“read” simultaneously throughout the 
cortex. Many of the sensors will be im- 
planted in the motor cortex, the region of 
the frontal lobe more readily associated 
with the generation of the motor pro- 
gram that is normally downloaded to the 
spinal cord, from which neurons directly 
control and coordinate the work of our 
muscles. (Some neuroscientists believe 
that this interaction between mind and 
muscle may be achieved through a nonin- 
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vasive method of recording brain activity, 
like EEG, but that goal has yet to be prac- 
tically achieved.) 

Gary Lehew in my group at Duke has 
devised a new type of sensor: a recording 
cube that, when implanted, can pick up 
signals throughout a three-dimensional 
volume of cortex. Unlike earlier brain sen- 
sors, which consist of flat arrays of micro- 
electrodes whose tips record neuronal 
electrical signals, Lehew’s cube extends 
sensing microwires up, down and sideways 
throughout the length of a central shaft. 

The current version of our recording 
cubes contains up to 1,000 active record- 
ing microwires. Because at least four to 
six single neurons can be recorded from 
each microwire, every cube can poten- 
tially capture the electrical activity of be- 
tween 4,000 to 6,000 neurons. Assuming 
that we could implant several of those 
cubes in the frontal and parietal corti- 
ces—areas responsible for high-level con- 
trol of movement and decision making— 
we could obtain a simultaneous sample 
of tens of thousands of neurons. Accord- 
ing to our theoretical software modeling, 
this design would suffice for controlling 
the flexibility of movement required to 
operate an exoskeleton with two legs and 
to restore autonomous locomotion in our 
patients. 

To handle the avalanche of data from 
these sensors, we are also moving ahead 
on making a new generation of custom- 
designed neurochips. Implanted in a pa- 
tient’s skull along with the microelec- 
trodes, they will extract the raw motor 
commands needed to manipulate a whole- 
body exoskeleton. 

Of course, the signals detected from 
the brain will then need to be broadcast to 
the prosthetic limbs. Recently Tim Han- 
son, a newly graduated Ph.D. student at 
Duke, built a 128-channel wireless record- 
ing system equipped with sensors and 
chips that can be encased in the cranium 
and that is capable of broadcasting record- 
ed brain waves to a remote receiver. The 
first version of these neurochips is cur- 
rently being tested successfully in mon- 
keys. Indeed, we have recently witnessed 
the first monkey to operate a brain-ma- 
chine interface around the clock using 
wireless transmission of brain signals. We 
filed in July with the Brazilian govern- 
ment for permission to use this technolo- 
gy in humans. 

For our future soccer ball kicker, the 


CHRONOLOGY 


The Long 


Road to 


Brain-Controlled Prosthetics 


Replacement limbs have existed for millennia—a rational response to the need 
to address war wounds or other types of trauma and birth defects. Today the 
technology is so sophisticated that an artificial limb can be controlled by electrical 


signals channeled directly from the brain. 


1500-1000 B.c. 

FIRST HISTORICAL REFERENCE 
AHindu holy book written during this period 
mentioned Vishpala, who had a leg amputation after 
a wound sustained during battle. She had the limb 
replaced with an iron version that let her walk and 
return to her troops. 


FOURTH CENTURY B.c. 
ANCIENT ARTIFACT 
One of the oldest artificial limbs discovered—a copy 

of which is shown here—was dug up in southern Italy 
in 1858. Fabricated in about 300 B.C., it was made of 
copper and wood and designed, it appears, for a below- 
knee amputee. 


14TH CENTURY 
GUNS AND AMPUTATIONS 

The arrival of gunpowder at the European battlefront 
greatly amplified the number of injuries sustained by 
soldiers. In response, in the 16th century AMBROISE 
PARE, the royal surgeon for several French kings, 
developed techniques to attach both upper and lower 
limbs to patients and reintroduced the use of ligatures 
to tie off blood vessels. 


1861-1865 


CIVIL WAR 

The War between the States resulted in many amputations. 
One person affected was BRIGADIER GENERAL 
STEPHEN JOSEPH MCGROARTY, who lost 
an arm. An influx of government funding and the avail- 
ability of anesthetics that allowed for longer operations 
improved prosthetic technology during this era. 


1963 PRIMITIVE BRAIN INTERFACE 
José Manuel Rodriguez Delgado implanted a radio- 
controlled electrode in the caudate nucleus deep 
ina bulls brain and stopped the animal dead in its 
tracks by pressing a button on a remote transmitter; 
his device was a predecessor to contemporary brain- 
machine interfaces. 


1969 PIONEERING EXPERIMENTS 
Eberhard Fetz of the University of Washington 
performed a study in which monkeys were trained 
to activate electrical signals in their brain to control 
the firing of a single neuron, duly recorded by 

a metal microelectrode. 
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1980s 

LISTENING TO BRAIN WAVES 

Apostolos Georgopoulos of Johns Hopkins University 
discovered an electrical firing pattern in the motor 


neurons of rhesus macaques that occurred when they 


rotated their arm in a particular direction. 


EARLY 1990s PLUGGING IN 
John Chapin, now at S.U.N-Y. Downstate University, 
and Miguel A. L. Nicolelis introduced a technique 
that allowed for simultaneous recording of dozens 
of widely dispersed neurons using permanently 
implanted electrodes, thus paving the way for 
research on brain-machine interfaces. 


1997 BETTER MOVES ——_— 
The microprocessor-controlled C-LEG KNEE 
PROSTHESIS, which in its current version allows 
the wearer to turn on customized settings that can be 
used for activities such as bicycling, was introduced. 


1999-2000 GOOD FEEDBACK 

The Chapin and Nicolelis laboratories published 
the first description of a brain-machine interface 
operated by activity from rat brains, whereby 
the animals sensed the movement through a 
visual feedback signal. The following year the 
Nicolelis lab published the first study in which a 
monkey controlled the movements of a robotic 
arm using only brain activity. 


2008-2011 BLADE RUNNER ———— 
After failing to qualify for the 2008 Summer 
Olympics Games, OSCAR PISTORIUS 
swept the 2008 Summer Paralympic Games and 
then got to the 400-meter semifinals at the 2011 
International Association of Athletics Federations 
World Championships in Daegu, South Korea. 


2011 MONKEY THINK, AVATAR DO 
Nicolelis’s team at the Duke University Center 

for Neuroengineering demonstrated that a 
monkey was able to use thoughts to manipulate 
the movements of a software avatar. 


2012 FROM MY BRAIN 

TO MY ROBOT ARM 

John Donoghue of Brown University and his 
colleagues showed with their BRAINGATE 
NEURAL INTERFACE SYSTEM that 

a subject with a brain implant could manipulate 
a robotic arm to pick up a drink. 


2014 CYBORG OPENING KICK 
The Nicolelis lab intends to provide an 
exoskeleton for a handicapped teenager 


to make the first kick of the opening event 
of the World Cup in Brazil. 
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data from the recording systems will be 
relayed wirelessly to a small computer 
processing unit contained in a backpack. 
Multiple digital processors will run vari- 
ous software algorithms that translate 
motor signals into digital commands 
that are able to control moving parts, or 
actuators, distributed across the joints of 
the robotic suit, hardware elements that 
adjust the positioning of the exoskele- 
ton’s artificial limbs. 


FORCE OF BRAINPOWER 

THE COMMANDS will permit the exoskele- 
ton wearer to take one step and then an- 
other, slow down or speed up, bend over 
or climb a set of stairs. Some low-level ad- 
justments to the positioning of the pros- 
thetic hardware will be handled directly 
by the exoskeleton’s electromechanical 
circuits without any neural input. The 
space suit-like garment will remain flexi- 
ble but still furnish structural support to 
its wearer, a surrogate for the human spi- 
nal cord. By taking full advantage of this 
interplay between brain-derived control 
signals and the electronic reflexes sup- 
plied by the actuators, we hope that our 
brain-machine interface will literally car- 
ry the World Cup kicker along by force of 
willpower. 

The kicker will not only move but also 
feel the ground underneath. The exoskel- 
eton will replicate a sense of touch and 
balance by incorporating microscopic 
sensors that both detect the amount of 
force from a particular movement and 
convey the information from the suit 
back to the brain. The kicker should be 
able to feel that a toe has come in contact 
with the ball. 

Our decade-long experience with brain- 
machine interfaces suggests that as soon 
as the kicker starts interacting with this 
exoskeleton, the brain will start incorpo- 
rating this robotic body as a true exten- 
sion of his or her own body image. From 
training, the accumulated experience ob- 
tained from this continuous feeling of 
contact with the ground and the position 
of the robotic legs should enable move- 
ment with fluid steps over a soccer pitch 
or down any sidewalk. All phases of this 
project require continuous and rigorous 
testing in animal experiments before we 
begin in humans. In addition, all proce- 
dures must pass muster with regulatory 
agencies in Brazil, the U.S. and Europe to 
ensure proper scientific and ethical re- 
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view. Even with all the uncertainties in- 
volved and the short time required for 
the completion of its first public demon- 
stration, the simple idea of reaching for 
such a major milestone has galvanized 
Brazilian society’s interest in science in 
ways rarely seen before. 


REMOTE CONTROL 

THE OPENING KICKOFF of the World Cup—or 
a similar event, say, the 2016 Olympic 
and Paralympic Games in Rio de Janeiro, 
if we miss the first deadline for any rea- 
son—will be more than just a one-time 
stunt. A hint of what may be possible 
with this technology can be gleaned from 
a two-part experiment already complet- 
ed with monkeys. As a prelude, back in 
2007, our research team at Duke trained 
rhesus monkeys to walk upright on a 
treadmill as the electrical activity of 
more than 200 cortical neurons was re- 
corded simultaneously. Meanwhile Gor- 
don Cheng, then at ATR Intelligent Ro- 
botics and Communication Laboratories 
in Kyoto, built an extremely fast Internet 
protocol that allowed us to send this 
stream of neuronal data directly to Kyo- 
to, where it fed the electronic controllers 
of CB1, a humanoid robot. In the first 
half of this across-the-globe experiment, 
Cheng and my group at Duke showed 
that the same software algorithms devel- 
oped previously for translating thoughts 
into control of robotic arms could also 
convert patterns of neural activity in- 
volved in bipedal locomotion to make 
two mechanical legs walk. 

The second part of the experiment 
yielded a much bigger surprise. As one of 
our monkeys, Idoya, walked on the tread- 
mill in Durham, N.C., our brain-machine 
interface broadcast a constant stream of 
her brain’s electrical activity through 
Cheng’s Internet connection to Kyoto. 
There CB1 detected these motor com- 
mands and began to walk as well, almost 
immediately. CB1 first needed some sup- 
port at the waist, but in later experi- 
ments it began to move autonomously in 
response to the brain-derived commands 
generated by the monkey on the other 
side of the globe. 

What is more, even when the tread- 
mill at Duke stopped and Idoya ceased 
walking, she could still control CBI’s leg 
movements in Kyoto by merely observ- 
ing the robot’s legs moving on a live vid- 
eo feed and imagining each step CBI 


should take. Idoya continued to produce 
the brain patterns required to make CB1 
walk even though her own body was no 
longer engaged in this motor task. This 
transcontinental brain-machine inter- 
face demonstration revealed that it is 
possible for a human or a simian to read- 
ily transcend space, force and time by 
liberating brain-derived commands from 
the physical limits of the biological body 
that houses the brain and broadcasting 
them to a man-made device located far 
from the original thought that generated 
the action. 

These experiments imply that brain- 
machine interfaces could make it possi- 
ble to manipulate robots sent into envi- 
ronments that a human will never be able 
to penetrate directly: our thoughts might 
operate a microsurgical tool inside the 
body, say, or direct the activities of a hu- 
manoid worker trying to repair a leak at a 
nuclear plant. 

The interface could also control tools 
that exert much stronger or lighter forces 
than our bodies can, thereby breaking free 
of ordinary constraints on the amount of 
force an individual can exert. Linking a 
monkey’s brain to a humanoid robot has 
already done away with constraints im- 
posed by the clock: Idoya’s mental trip 
around the globe took 20 milliseconds— 
less time than was required to move her 
own limb. 

Along with inspiring visions of the far 
future, the work we have done with mon- 
keys gives us confidence that our plan 
may be achievable. At the time of this 
writing, we are waiting to see whether 
the International Football Association 
(FIFA), which is in charge of organizing 
the ceremony, will grant our proposal to 
have a paraplegic young adult participate 
in the opening ceremony of the inaugural 
game of the 2014 World Cup. The Brazil- 
ian government—which is still awaiting 
FIFA’s endorsement—has tentatively sup- 
ported our application. 

Bureaucratic difficulties and scientific 
uncertainties abound before our vision 
is realized. Yet I cannot stop imagining 
what it will be like during the brief but 
historic stroll onto a tropical green soccer 
pitch for three billion people to witness a 
paralyzed Brazilian youth stand up, walk 
again by his or her own volition, and ulti- 
mately kick a ball to score an unforgetta- 
ble goal for science, in the very land that 
mastered the beautiful game. 


MORE TO EXPLORE 


Controlling Robots with the Mind. 
Miguel A. L. Nicolelis and John K. 
Chapin in Scientific American, Vol. 287, 
No. 4, pages 46-53; October 2002. 
Cortical Control of a Prosthetic 
Arm for Self Feeding. Mee! Velliste 
et al. in Nature, Vol. 453, pages 1098- 
1101; June 19, 2008. 

Beyond Boundaries: The New Neu- 
roscience of Connecting Brains with 
Machines—and How It Will Change 
Our Lives. Miguel Nicolelis. St. Mar- 


tin’s Griffin, 2012. 


SCIENTIFIC AMERICAN 
ONLINE 

Inspect an exoskeleton 
prototype at ScientificAmerican. 
com/sep2012/exoskeleton 


September 2012, ScientificAmerican.com 63 


343 


344 


PUBLISHED: 10 JANUARY 2017 | VOLUME: 1| ARTICLE NUMBER: 0008 


comment 


Are we at risk of becoming 
biological digital machines? 


Miguel A.L. Nicolelis 


The brain can be viewed as an organic computer that can be reprogrammed to incorporate external 
elements, such as artificial tools. But is there a risk that our increasing reliance on digital devices, such 
as smartphones, could also be reprogramming our brains and blunting our human attributes? 


n Neal Stephenson’s science-fiction novel 

Snow Crash’, Hiro Protagonist, “a warrior 

prince in the metaverse” and one of the 
most skilful hackers of his generation, 
is on a mission to save humanity from 
its true neurophysiological doomsday. 
Hiro’ herculean task is to try to stop a 
new type of virus, Snow Crash, from 
infecting the minds of people all over the 
world and transforming them into mere 
biological automata, devoid of any trace 
of real consciousness, free will, agency 
or individuality. Once uploaded into the 
brainstem, Snow Crash insidiously hacks 
the basic machine code that runs in our 
subcortical limbic system, leading to a 
complete halt of neocortical subroutines. 
All this neurobiological mayhem turns 
the infected person into a perfect zombie, 
someone incapable of reasoning critically 
or maintaining his/her natural state 
of consciousness. 

Are contemporary humans running 

a similar risk? Could our constant 
reciprocal interaction with digital logic 
(through laptops, tablets, smartphones, 
all the way to highly immersive virtual 
reality environments), particularly when it 
leads to powerfully hedonic experiences, 
result in the slow compromise or even 
elimination of some of the behaviours and 
cognitive aptitudes that represent the most 
exquisite and cherished attributes of the 
human condition? Attributes such as our 
multifaceted social skills, empathy, linguistic 
semantics, aesthetic sense, artistic expression, 
intuition, creativity and the ability to 
improvise solutions to novel contingencies, to 
name just a few. In other words, could opting 
for the fast lane of the never-ending highway 
to full digital immersion and automation — 
an obvious current trend in our modern 
society — produce a reduction in human 
cognitive capabilities that resembles, even if 
only remotely, the less-than-enticing fate of 
those infected by Snow Crash? 


CREATIVE-IDEA/DIGITALVISION VECTORS/GETTY 


The brain as an organic computer 
Since Alan Turing’s original proposition in 
the late 1930s of what became known as the 
universal Turing machine, the debate on 
whether an animal brain, and in particular 
the human central nervous system, can 
be considered a digital computer has 
raged. Artificial-intelligence researchers 
who believe that digital machines can 
acquire human-like intelligence have 
no qualms in accepting this view at face 
value. Surprisingly, a few neuroscientists 
tend to agree with this position and even 
suggest that animal brains, including ours, 
can be fully simulated in a digital device. 
On the other hand, many philosophers, 
neuroscientists and physicists have staunchly 
disagreed with this proposition, labelling it 
as nothing more than wishful thinking or, 
worse, a form of mystical belief. 

Recently, Ronald Cicurel and 
I? summarized the evolutionary, 
neurobiological, computational and 
mathematical arguments that challenge the 
notion that the operation of the human brain 
can be reduced to the algorithmic nature 
of Turing machines. Instead, we proposed 
that higher-order brains, including ours, 
constitute a different type of computation 
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device altogether, which we referred to as an 
organic computer. One of the distinctions 
between digital and organic computers is 
that higher animal brains use a recursive 
mix of analogue and digital processing to 
compute and are not only capable of, but 
highly specialized in, handling semantic- 
rich information, which differs markedly 
from the classic syntactically rigid type of 
information on which digital machines rely. 
Furthermore, in our brains, the concepts of 
software and hardware cannot be dissociated 
from each other; both are welded together 
in the very organic matter that defines the 
organic computer. As such, brains utilize the 
coherent blending of their multiple levels of 
organization — from intracellular protein 
complexes, to cell membranes, to individual 
whole cells, to neuronal circuits, all the way 
to the entire system — to compute. 


Programming the human brain 

But there is an important caveat. Even 
though organic computers cannot be 
reduced to a Turing machine, they can be 
programmed by many distinct biological 
processes. For instance, at the most basic 
level, the expression of numerous genes in 
the human genome, selected by a multitude 
of evolutionary events, interact as part of a 
‘genetic programme’ that is responsible for 
assembling the brain’s natural 3D structure 
during prenatal and postnatal life. This 
genetic programming guarantees that our 
brain’s initial physical configuration reflects 
the same basic neural architecture that 
evolved in anatomically modern humans 
about 100,000 years ago. Once we are 
born, the brain’s programming continues 
as we grow and learn, through interactions 
with our social and physical environment. 
Continuous immersion in human culture 
and its cornucopia of social interactions 
further programmes the central nervous 
system. Indeed, the so-called social brain 
hypothesis** proposes that the tremendous 
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growth in neocortical volume, experienced 
during the evolution of anthropoid primates 
into Homo sapiens, is intimately related 

to the concomitant increase in social 
complexity that these species experience 

in their daily routine living in groups. This 
correlation can be readily visualized when 
the cortical volume of several primates is 
plotted against the typical social group size 
of each species. Accordingly, the significant 
growth in human cortical volume would 
explain why our species developed the 
capacity to handle close social interactions 
in groups of about 150 individuals, while 
chimpanzees and baboons limit their bands 
to 50-55 individuals’. 


Reprogramming the human brain 

But this is not all. There are other possible 
ways to programme or even reprogramme 
the primate brain that we carry between our 
ears. A decade and a half of intense basic 
and clinical research with brain/machine 
interfaces (BMIs) has clearly demonstrated 
that the primate brain, including our own, 
is capable of assimilating artificial tools — 
like robotic or even avatar limbs — as 
extensions of the brain-built representation 
of the subject’s own body’, through the 
phenomenon of cortical plasticity. This 
means that neuronal space is dedicated to 


representing the operation of artificial tools’. 


These findings, together with other results 
from various studies, indicate that our 
sense of self can be readily reprogrammed 
to incorporate external elements. We have 
recently documented such an extension 
of the sense of self in severely paralyzed 
patients trained to employ a BMI to 
control the movements of an avatar body’. 
Indeed, such reprogramming through 
cortical, and probably subcortical, plastic 
changes might also account for the partial 
neurological clinical recovery experienced 
by these patients’. 

‘That brings us back to the central issue 
of this brief Comment article. Even though 
the brain cannot be reduced to a digital 
machine, could the human brain simply 
assimilate and begin to mimic the rigid 
binary logic and algorithmic mode of 
operation of digital machines due to the 
growing overexposure to digital devices 
and the hedonic response triggered by these 
interactions, and become a biological digital 
system? Given the ominous introduction 
of digital automation in almost all aspects 
of human life, it is not surprising that a 
large literature exists describing the impact 


of the introduction of digital systems on 
human behaviour and mental skills. In his 
book’, The Glass Cage: Automation and Us, 
Nicholas Carr reviews some of these studies, 
showing that continuous exposure to digital 
systems can have profound effects on 
human performance, from the flying skills 
of airplane pilots, to the pattern recognition 
ability of radiologists, to the broad sense of 
creativity of architects. 


In our brains, the concepts 
of software and hardware 
cannot be dissociated from 
each other. 


If had to propose a hypothesis to account 
for all these findings, I would volunteer 
the notion that passive immersion in the 
digital systems of modern airplanes (in the 
case of pilots), digital imaging diagnostics 
(radiologists) and computer-assisted design 
(architects) may gradually curtail the range 
and acuity of some mental functions and 
cognitive skills, such as creativity, insight and 
the ability to solve novel problems. 

A study by Betsy Sparrow et al.? showed 
that when people believe that a series 
of statements that they have been asked 
to remember will be stored online, they 
perform worse than a control group that 
relies only on their own biological memory 
to remember the statements. This suggests 
that subcontracting some simple mental 
searches to Google may, after all, reduce 
our own brain’s ability to store and recall 
memories reliably. 

The impact of online social media on our 
natural social skills is another area in which 
we may be able to measure the true effects of 
digital systems on human behaviour. In her 
book", Alone Together: Why We Expect More 
from Technology and Less from Each Other, 
Sherry Turkle describes her experience 
interviewing teenagers and adults who 
are heavy users of texting, social media 
and other online virtual environments. 

An intense presence on social media and 
virtual reality environments can produce 
significant anxiety, a reduction in real 

social interactions, lack of social skills and 
human empathy, and difficulties in handling 
solitude. Moreover, symptoms and signs of 
addiction to virtual life are often reported 
almost casually by some of the interviewees. 

After reading Turkle’s interviews, I 
began wondering whether the new ‘always 


connected’ routine is overtaxing our cerebral 
cortex by dramatically expanding the 
number of people with whom we can closely 
communicate, almost instantaneously, via the 
multitude of social media outlets available 
on the internet. Instead of respecting the 
group size limit (about 150 individuals) 
afforded by our cortical volume, we are now 
in continuous contact with a group of people 
that could far exceed that neurobiological 
limit. What are the consequences of this 
cortical overtaxing? Anxiety, attention, 
cognitive and even memory deficits? 


Homo digitalis 
Is the above scenario something we should 
pay attention to? I think so. If not because of 
the potential impact on the mental health of 
this and future generations, but also because 
of the far-reaching consequences of our 
increasing interaction with digital systems. 
For example, at the far limit, I can conceive 
that this staggering expansion in our online 
social connectivity is capable of providing 
a completely new type of selective pressure 
that may, eventually, bias the evolutionary 
future of our species. One may begin 
wondering whether the dawn of ‘Homo 
digitalis’ is upon us or, more surprisingly, 
whether he/she is already around, texting 
and tweeting without being noticed. 
Evidently, much more research will be 
needed to address these intriguing questions 
and the true impact of digital technology on 
shaping key functions of the human brain. 
Meanwhile, just in case we need his services 
in the near future, next time I meet Neal 
Stephenson, I will be sure to ask for Hiro 
Protagonist’s phone number and e-mail, 
Facebook, Twitter and LinkedIn addresses. 
That is, if I remember to ask. im 


Miguel A. L. Nicolelis is in the Department of 
Neurobiology, Box 3209 Duke University, Durham, 
North Carolina 27710, USA. 

e-mail: nicoleli@neuro.duke.edu 
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ABSTRACT: Spinal cord stimulation has been used 
for the treatment of chronic pain for decades. In 2009, 
our laboratory proposed, based on studies in rodents, 
that electrical stimulation of the dorsal columns of the 
spinal cord could become an effective treatment for 
motor symptoms associated with Parkinson’s disease 
(PD). Since our initial report in rodents and a more 
recent study in primates, several clinical studies have 
now described beneficial effects of dorsal column 
stimulation in parkinsonian patients. In primates, we 
have shown that dorsal column stimulation activates 
multiple structures along the somatosensory pathway 
and desynchronizes the pathological cortico-striatal 
oscillations responsible for the manifestation of PD 


me 


% 


symptoms. Based on recent evidence, we argue that 
neurological disorders such as PD can be broadly 
classified as diseases emerging from abnormal neuro- 
nal timing, leading to pathological brain states, and 
that the spinal cord could be used as a “channel” to 
transmit therapeutic electrical signals to disrupt these 
abnormalities. © 2017 International Parkinson and 
Movement Disorder Society 


Key Words: spinal cord stimulation; Parkinson’s 
disease; neuronal oscillations; deep brain stimulation; 
dopamine 
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Parkinson’s disease (PD) is a chronic, progressive 
neurodegenerative disease’? resulting from the loss of 
dopamine-producing brain cells in the substantia nigra 
pars compacta.* Patients suffering from PD experience 
progressive motor impairments,’ which include tremor, 
rigidity, bradykinesia, and gait instability. It is esti- 
mated that approximately 0.4 to 1 million Americans 
live with PD.°** Although dopamine replacement ther- 
apy, through administration of the dopamine precursor 
L-3,4-dihydroxyphenylalanine (1-dopa), _ effectively 
ameliorates PD symptoms in the early stages of the dis- 
ease,’ prolonged use of the drug results in dose-limiting 
side effects,*” reduced efficacy,'®!' and complications 
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such as L-dopa—induced dyskinesia.!”!? In this context, 
additional therapeutic strategies, such as deep brain 
stimulation (DBS), that have proven to be effective for 
treating the main PD motor symptoms have attracted 
considerable attention in the past few decades.'*17 

Despite its unquestionable success, there are some 
disadvantages associated with DBS. First, it requires a 
highly invasive and expensive surgical procedure, which 
targets a very small structure deep in the brain.'®!? 
Because of this, the neurosurgical procedure for DBS 
comes associated with a 1.1% risk of death during the 
operation and 1.2% to 15.2% risk of other major com- 
plications.” This means that only very experienced 
functional neurosurgeons can perform the procedure. 
Altogether, these factors, in particular the procedure’s 
invasiveness, limit the procedure to just a small fraction 
of severely ill PD patients: 1.6% to 4.5%.°*° 

Another important issue is that, although DBS pro- 
duces a very significant improvement of the patient’s 
tremor and rigidity, it is less effective in treating those 
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who suffer primarily from bradykinesia or gait insta- 
bility and “freezing.”'*?775 Thus, although DBS 
greatly improves the quality of life in patients with 
advanced PD by addressing the disease’s cardinal 
motor symptoms while reducing levodopa-induced 
dyskinesias, locomotion impairment can be extremely 
disabling and severely affect the quality of life of PD 
patients, even when the primary symptoms are con- 
trolled by this surgical therapy. Furthermore, if one 
considers that standard treatments for bradykinesia 
and freezing, such as physical therapy, are often inef- 
fective in PD, it becomes clear that novel therapeutic 
approaches are required to address postural instability 
and gait disturbance in PD patients.7*7?°° 

In this context, a significantly less invasive method—epi- 
dural dorsal column stimulation (DCS)—has been sug- 
gested by our laboratory as an alternative approach for 
symptomatic treatment of PD. DCS is a well-established 
therapy for the treatment of chronic neuropathic pain. In 
the United States, DCS has been approved by the Food and 
Drug Administration for treating chronic low back and 
limb pain.?’ Back in 2009, our laboratory reported on the 
initial experimental demonstrations of the potential thera- 
peutic benefit of DCS in multiple rodent models of PD.*” 
Initially, we demonstrated an acute effect of DCS on akine- 
sia and bradykinesia in rodents.” Later, the same effect 
was reproduced in nonhuman primate models.*? Further 
studies revealed both a long-term motor improvement and 
a neuroprotective effect on the rat nigrostriatal dopaminer- 
gic system of rodents, following chronic DCS.** 

Following these initial studies, other laboratories 
have independently validated our animal findings.**°° 
In parallel, during the past 7 years, several indepen- 
dent clinical studies in PD patients with abnormal pos- 
ture and gait disturbances have demonstrated positive 
results with DCS.°7*8 

The central goal of this short review is to cover the 
recent literature that supports our initial contention 
that DCS can become a useful new therapy for PD, 
particularly for those patients who cannot benefit 
from DBS because their main symptoms are related to 
gait dysfunction. 


Spinal Cord Stimulation 


Electrical stimulation of the dorsal funiculus of the 
spinal cord was first used for the treatment of chronic 
pain by Shealy and colleagues at Case Western 
Reserve University in 1967.** Since then it has become 
a treatment for different pain syndromes such as pain 
from failed back surgery syndrome or intractable low 
back pain?'*°-*° and also investigated for numerous 
novel pain syndromes.*”*? The rationale for using 
DCS in pain treatments was initially provided by the 
classic gate control theory of pain proposed originally 
by Melzack and Wall.°° According to the theory, by 
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activating the large diameter, myelinated, non- 
nociceptive fibers of the dorsal funiculus of the spinal 
cord, DCS would produce a closing of the “gate” at 
the dorsal horns, leading to suppression of the noxious 
stimuli carried by the much smaller unmyelinated C 
fibers that transmit nociceptive information that con- 
tributes to the genesis of pain. Subsequently, other 
potential mechanisms for the suppression of pain by 
DCS were uncovered, including the following: 


1. Paresthesia triggered by the activation of supra- 
spinal circuits produced by orthodromic activa- 
tion of dorsal columns (DCs),°!>* 

2. Changes in the local transmitter systems and the 
suppression of dorsal horn neurons in the case of 
neuropathic pain,*°* and 

. The balance of the oxygen supply and demand 
by inhibition of sympathetic activity in the case 
of ischemic pain.?!*° 


ies) 


Interestingly, over the years and in parallel with the 
growing use of this approach for chronic pain, several 
reports of the potential therapeutic effects of DCS in 
improving motor symptoms in patients suffering from 
various motor disorders, such as dystonia,°®°? multiple 
sclerosis,°° nonparkinsonian tremor,°'** and painful leg 
and moving toes syndrome,°*™ have appeared in the 
literature.°° 

Although the exact neurophysiological mechanisms 
involved in relieving the symptoms in these motor disor- 
ders by DCS have not been elucidated, computational 
modeling studies have shown that epidural electrical 
stimulation of the spinal cord, when applied within the 
therapeutic range, usually activates DC fibers and the 
dorsal roots in the vicinity of the stimulating cathode. 
In addition, functional magnetic resonance imaging dur- 
ing DCS application in patients revealed clear modula- 
tion of activity in cortical structures, such as primary and 
secondary somatosensory cortices ($1 and $2), prefrontal 
cortex, cingulate cortex, insula, and thalamus.°”° 

DCS has also been shown to cause changes in c-fos 
expression in supraspinal structures, which in turn 
may lead to long-term sustained effects.’”°’! Both of 
these findings are consistent with the neurophysiologi- 
cal and immunohistochemical results obtained in our 
animal studies with DCS, which show that this 
approach disrupts pathological oscillations around the 
cortical-basal ganglia circuitry while inducing a neuro- 
protective effect on the dopaminergic neurons of the 
nigrostriatal system.** 


Low-Frequency Neuronal 
Oscillations Are Correlated With PD 
Symptoms 


The basal ganglia is involved in the execution of 
goal-directed behavior in conjunction with the cortex 
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via an extensive circuitry formed by multiple structures 
that share feedforward and feedback pathways.”* Mul- 
tiple clinical studies in parkinsonian patients have 
detected the presence of pathologically high levels of 
synchronous oscillatory neuronal and local field poten- 
tial activity in the basal ganglia, which is particularly 
prominent in the beta frequency range of 15 to 30 
Hz.’*”> Animal models of PD also exhibit similar path- 
ological activity. For example, monkeys treated with 
the toxin 1-methyl-4-phenyl-1,2,3,6-tetrahydropyridine 
(MPTP) show a significant increase in the fraction of 
basal ganglia neurons with oscillatory frequency in the 
3 to 8 Hz and 8 to 15 Hz ranges. Often this neuronal 
activity is correlated with the animal’s tremor.’%’’ 
Multiple reports using the MPTP model have identified 
neuronal synchronous oscillatory activity in the subtha- 
lamic nucleus (STN), internal globus pallidus, external 
globus pallidus, and substantia nigra pars reticulata 
neurons.’°’® Similarly, studies involving the 6-hydroxy- 
dopamine (6-OHDA) rat model have also demonstrated 
the presence of beta frequency (15-30 Hz) oscillations 
in STN and substantia nigra pars reticulata.””"*! 

Sharott and colleagues*” demonstrate that the power 
of beta frequency oscillations and their coherence 
between the STN and cortex was significantly reduced 
after the administration of apomorphine (a dopamine 
agonist) in rats, shifting the peak coherence to higher 
frequencies, a phenomenon we have also observed after 
t-dopa administration in dopamine-depleted mice.** 
This effect coincides with previous observations in the 
basal ganglia areas of parkinsonian patients after dopa- 
minergic medication.°*** In monkeys, dopamine 
replacement therapy®* and STN inactivation through 
DBS*° significantly ameliorate MPTP-induced tremor 
while reducing prominent 8 to 20 Hz oscillations in the 
basal ganglia. DBS has shown to effectively improve 
PD symptoms in humans while attenuating synchro- 
nous beta band activity in the cortico-basal ganglia net- 
work.®””° It is, however, important to note that the 
reduction in beta band activity after medication as well 
as DBS is often correlated with the improvement in 
motor performance, suggesting that the attenuation of 
oscillatory and synchronous neural activity has a thera- 
peutic effect on PD symptoms.?'”? 


Spinal Cord Stimulation Mechanism 
for PD 


For the past 7 years, our laboratory has studied the 
effects of DCS on PD symptoms in rodents and non- 
human primates. Based on previous evidence that elec- 
trical stimulation of the peripheral afferents of a 
major somatosensory pathway could be used to induce 
potent cortical desynchronization,’* we proposed that 
high-frequency synchronous activation of the DC 
fibers could lead to cortico-striatal desynchronization, 
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thereby causing the alleviation of PD symptoms. Our 
studies revealed that DCS treated animals exhibited 
much higher levels of locomotion than dopamine- 
depleted mice, 6-OHDA lesioned rats, and 6-OHDA 
lesioned marmosets.**** Increased oscillatory power 
in the 1.5 to 4 Hz and 10 to 15 Hz bands and 
decreased power in 25 to 55 Hz bands was observed 
during the dopamine-depleted state in mice.*? DCS 
created a shift in spectral power from lower to higher 
frequencies (Fig. 1) and produced a neuronal firing 
pattern similar to the one observed prior to locomo- 
tion initiation.°* The fraction of neurons in the M1 
and striatum that were entrained to the Local Field 
Potential (LFP) activity also dropped considerably 
after the application of DCS. Although the onset of 
locomotion on DCS application was delayed by a few 
seconds, the changes in neuronal activity were almost 
spontaneous. These findings suggested that DCS cre- 
ated a brain state permissible for locomotion onset. 
Thereafter, our study using a marmoset monkey 
model of PD revealed that DCS indeed alleviates aki- 
nesia and restores the pathological brain state, defined 
by abnormal neuronal bursting and oscillatory activ- 
ity, to normalcy by altering the functional coupling 
between multiple areas in the cortico-basal ganglia- 
thalamic loop.*? During the induced PD state, we 
observed a significant increase in functional coherence, 
in the 8 to 15 Hz range, between multiple neural 
structures that belong to the cortical-basal ganglia cir- 
cuitry. During DCS, however, this enhanced coher- 
ence—which can also be seen as enhanced functional 
connectivity—was reduced, an effect similar to what 
was observed after 1-dopa administration. DCS 
resulted in the suppression of 8 to 20 Hz beta band 
LFP power as well as the beta rhythmicity of neurons 
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FIG. 1. Dorsal column stimulation (DCS) restores locomotion and 
desynchronize corticostriatal activity (reprinted with permission; originally 
published in Fuentes et al.°”). Average spectrograms of striatal local field 
potentials (LFP) (in dopamine-depleted mice) recorded around 300-Hz 
stimulation events (yellow bar) shown in top row indicates the spectral 
power (denoted by black trace) shifting to higher frequencies, middle 
row showing LFP power standardized to first 240 seconds demonstrates 
desynchronization, whereas the bottom row shows increased locomo- 
tion during stimulation “ON” period. 
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FIG. 2. Dorsal column stimulation (DCS) reverses weight loss, restores motor function, and protects dopaminergic neurons in a 6-hydroxydopamine 
(6-OHDA) model of Parkinson’s disease (reprinted with permission; originally published in Yadav et al.°“). (A) Changes in body weight after bilateral 
intrastriatal 6-OHDA lesion with or without DCS treatment. Lesioned, nontreated rats (n = 8) suffered sustained weight loss with little to no recovery. 
Lesioned rats with DCS treatment (n = 7, 30 minutes, 333 Hz continuous DCS during 30 minutes twice a week, starting 7th day, black arrow) recov- 
ered body weight significantly faster than nontreated rats. (B) Lesioned rats develop crouched posture resulting in shorter major axis length. DCS 
treatment restores posture significantly faster than nontreated rats. (C) Representative immunostaining for tyrosine hydroxylase in substantia nigra 
pars compacta (SNc); scale bar = 500 um. There was a significant difference between the tyrosine hydroxylase (TH) levels of 6-OHDA and 6- 


OHDA+DCS groups in the SNc. VTA, ventral tegmental area. 


in the cortico-basal ganglia-thalamic loop. Detailed 
analysis revealed that DCS activation of the primary 
somatosensory cortex via the ventral posterolateral 
nucleus of the thalamus induces a potent disruption of 
beta oscillations in multiple cortical and subcortical 
neural structures via a phase reset mechanism, where 
the LFP oscillations get phase locked to the incoming 
DCS pulses.?* This finding corroborated our original 
hypothesis that DCS desynchronizes corticostriatal 
oscillations, following the activation of the DC-medial 
lemniscal pathway. 


Preliminary Evidence of 
Neuroprotection 


In a subsequent series of experiments (Fig. 2), we 
investigated the long-term effects of DCS on animal 
body weight, motor symptoms, and_ survival of 
nigrostriatal dopaminergic neurons in a chronic rat 
model of PD using 6-OHDA lesioning.** This study 
revealed that the application of DCS at regular inter- 
vals led to a progressive improvement in the observed 
motor impairment, such as gait and postural instabil- 
ity, and an accelerated recovery from weight loss. 
When compared with untreated control animals, 
motor improvement was accompanied by a higher 
density of dopaminergic innervation in the striatum 
and higher neuronal cell counts in the substantia 
nigra pars compacta (SNc) of DCS-treated rats. These 
results suggest that DCS applied in 6-OHDA-treated 
rats induced both a functional and structural recov- 
ery in the nigrostriatal circuitry. Remarkably, such a 
neuroprotective effect was achieved by delivering 
DCS for 30 minutes, twice a week only, suggesting 
that better effects could be produced by more fre- 
quent therapy. These results have now _ been 
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replicated independently by another laboratory,*° 
which reported that 1 hour/day of DCS, delivered for 
16 consecutive days, resulted in the improvement of 
PD symptoms and the significant preservation of 
tyrosine hydroxylase (TH) fibers in the striatum and 
TH-positive neurons in the SNe. Although the cellu- 
lar mechanism underlying this putative long-term 
neuroprotective action of DCS remains to be investi- 
gated, Shinko and colleagues*® also reported that 
DCS treatment upregulated the levels of vascular 
endothelial growth factor, a finding that could 
account, at least partially, for the observed neuropro- 
tective effects. Support for this idea comes from stud- 
ies in which intrastriatal injections of glial-derived 
nerve growth factor (GDNF) and brain-derived neu- 
rotrophic factor (BDNF) have shown significant pro- 
tection or restoration following 6-OHDA or MPTP 
lesions.” ?8 Comparative analysis revealed that 
GDNF was more effective than BDNF for correcting 
behavioral deficits and protecting nigrostriatal DA 
neurons.” Spieles-Engemann and colleagues'”’ have 
shown that STN DBS also exerts a neuroprotective 
effect on the SN neurons in a 6-OHDA rodent model 
of PD. A follow-up study from the same group 
revealed that this effect could be mediated by the 
increased levels of BDNF in the nigrostriatal system 
and the primary motor cortex.'°! Shinko and col- 
leagues note that upregulation of vascular endothelial 
growth factor may protect the nigrostriatal dopami- 
nergic system by improving microcirculation in the 
striatum as a result of enhanced glial proliferation 
and angiogenesis.°° Although the results showing 
neuroprotection in animal models using DCS or DBS 
are promising, there is no clinical evidence support- 
ing this claim, suggesting that further investigation 
using DCS in parkinsonian humans is necessary.!°* 
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Clinical Studies in PD Patients 


Following our original report showing that high- 
frequency DCS alleviated akinesia and bradykinesia in 
rodent models of PD, Thevathasan and colleagues!” 
investigated the effect of DCS on motor function in 2 
patients with advanced PD in a double-blind cross- 
over study. Patient 1 was stimulated at 130 Hz, 
whereas patient 2 received 300 Hz, and the following 
assessments were made 10 days postoperatively (see 
Table 1) using the Unified Parkinson’s Disease Rating 
Scale (UPDRS) motor scores, timed 10-meter walk, 
timed hand-arm movements, and timed lower limb in 
3 different conditions (off-stimulation, suprathreshold 
stimulation, and subthreshold stimulation). In this 
study, DCS failed to provide improvement in all of the 
assessments. However, after closely analyzing the 
experimental design of this study, we argued that this 
clinical study failed to show significant improvements 
in PD symptoms mainly because of major differences 
in both the geometry of stimulation electrodes used 
and the chosen location of the implant along the spi- 
nal cord.'°* Support for our contention followed soon 
after, when several clinical groups around the world 
began to report positive results with DCS in PD 
patients. For example, a 74-year-old man who had 
originally been implanted with a DCS electrode in the 
T9-T10 epidural area for treating back pain because 
of failed back surgery syndrome, reported increased 
tremor when the stimulator was turned off.*? Clinical 
examination revealed that the patient had developed 
PD-related tremor 8 years after the implantation of 
DCS. When DCS stimulation was turned on and a 
130-Hz stimulus was employed, the patient’s UPDRS- 
III scores (evaluated during 4 separate sessions) 
improved by almost 50%. Tremor in the upper and 
lower limbs showed significant improvement during 
the stimulation-on condition while axial symptoms of 
gait and posture were also improved. 

In another case report, a 68-year-old woman 
implanted with DCS electrodes for postlaminectomy 
syndrome exhibited dramatic clinical improvements in 
her parkinsonian tremor in concurrence with relief 
from leg pain when DCS stimulation was delivered at 
60 Hz.” Another 43-year-old woman with a PD his- 
tory was implanted with DCS electrodes at the C2 
level for neuropathic pain in her neck and upper 
extremities.*° After years 1 and 2 of DCS treatment, 
using a stimulation frequency of 40 Hz, her motor 
UPDRS scores showed significant progressive improve- 
ments, leading to a significant reduction in tremor, 
rigidity, gait imbalance, and neuropathic pain. 

A total of 15 patients (5 men and 10 women), rang- 
ing in age from 63 to 79 years (mean 71.1 years), 
with complaints of low back and leg pain received 
DCS implants in the thoracic area T7 to T12, 
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depending on pain localization.*” Overall, 7 of these 
patients had previously undergone DBS in the STN. 
At 1 year after surgery, the pain scores and the 
UPDRS scores were significantly improved. There was 
a clear relief of gait disorder and postural instability, 
indicating that DCS had a remarkable effect on the 
axial symptoms associated with advanced PD. Another 
study by Arii and colleagues*® involving 37 PD 
patients with camptocormia (a treatment-resistant pos- 
tural abnormality) demonstrated an immediate effect 
using repetitive trans-spinal magnetic stimulation in 
the lower thoracic to upper lumbar vertebral area. 
Landi and colleagues*’ also reported significant 
improvement in gait and postural stability in a patient 
with advanced PD who had previously undergone 
STN DBS. Another advanced PD patient implanted 
with DCS electrodes for chronic back pain showed no 
improvement in UPDRS scores or locomotion.*? Mit- 
suyama and colleagues’ reported pain relief as well 
as significant improvement in walking posture in 2 
patients with chronic lumbar pain and PD symptoms 
of abnormal posture, sagittal imbalance, and difficulty 
in locomotion. Similarly, 3 PD patients, who also suf- 
fered from intractable pain as a result of failed back 
surgery syndrome and lumbar canal stenosis, showed 
remarkable improvement in pain as well as rigidity 
and tremor reflected in their UPDRS III scores.'°” 
Finally, in a more recent study'°* conducted by a neu- 
rosurgical team at the University of Sao Paulo, Brazil, 
in 4 long-term PD patients who had been subjected to 
DBS 7 to 8 years prior but still exhibited serious prob- 
lems in locomotion including freezing, at the time of 
testing, 300 Hz DCS was found to induce a significant 
improvement in patient gait (Timed-Up-Go [TUG], 
50%-65%) and the 20-minute walking test (50%- 
60% on time and 65%-70% on the number of steps). 
A significant improvement in UPDRS III (38% OFF 
meds; P = .017), Freezing of Gait (FOG) (56%; P = 
.04), and Parkinson’s Disease Questionnaire (PDQ)-39 
(mobility 57%, P = .003; Activities of Daily Living 
(ADL) 28%, P = .02) was also documented. 

Based on the data described in the previous para- 
graph, it is fair to say that, 7 years after our initial 
report in rodents, a beneficial clinical effect of DCS on 
alleviating multiple parkinsonian symptoms has been 
consistently reported by several independent groups. 
These preliminary clinical studies validated our origi- 
nal proposal that DCS should be further examined as 
a potential new therapy for PD, particularly in cases 
involving gait problems and freezing. Encouraging as 
they are, however, it is important to emphasize that 
these initial clinical results involved a small and het- 
erogeneous number of patients, tested in an open-label 
and heterogeneous fashion. For example, in 1 case the 
patients were >70 years old, whereas in 2 other cases 
they were aged 43 and 68 years.*°*7'°3 Studies varied 
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FIG. 3. Illustrative model of the motor circuitry involving the direct and 
indirect pathways of basal ganglia, thalamus, and cortex and also the 
pathways from the postural and gait control with projections from the 
basal ganglia to the pedunculopontine nucleus and the spinal cord. 
(adapted with permission; originally published in de Andrade et al.'°). 
PPN, pedunculopontine nucleus; Str, striatum; GPe, globus pallidus 
external segment; GPi, globus pallidus internal segment; STN, subtha- 
lamic nucleus; SNc, substantia nigra pars compacta; SNr, substantia 
nigra pars reticulate. 


in the type of electrodes used, stimulation parameters 
selected, number of sessions performed, and timeline 
of symptom evaluation. Moreover, most patients also 
exhibited a broad range of pain symptoms in addition 
to PD symptoms. Although most patients had relief 
from symptoms of pain, Thiriez and colleagues® 
argue that the effect of DCS on PD motor symptoms 
in these multiple studies might depend on the patients’ 
initial quality of response to L-dopa therapy. Thus, 
although the initial human results seem promising 
(summarized in Table 1), large, randomized, double- 
blind, clinical studies need to be performed to truly 
assess the potential therapeutic effects of DCS in PD 
and other motor disorders. 

It is also important to highlight that the effect of 
DCS on axial symptoms of PD was noticeable in more 
recent clinical studies.2”°"*° The role of the peduncu- 
lopontine nucleus (PPN)—a part of the mesencephalic 
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locomotor region in the brain stem in the pathophysi- 
ology of PD—has gained considerable interest in 
recent times. Similar to striatal lesions, PPN lesions in 
rhesus monkeys have resulted in parkinsonian symp- 
toms of bradykinesia, hypokinesia, and flexed pos- 
ture.!°?!!° Stimulation of the PPN in PD patients has 
resulted in significant improvements in axial symp- 
toms, such as gait imbalance and postural instability, 
suggesting that DCS may also exert its clinical effect 
by modulation of PPN activity.''!''> Alternatively, 
other studies have highlighted the moderate response 
to PPN-DBS during longer evaluation periods and its 
inadequacy as an alternate target for DBS when not 
used in conjunction with STN-DBS.?”!!*!'5 However, 
the fact that PPN has reciprocal connections with the 
cortex, thalamus, basal ganglia, and spinal cord''® 
implies that modulation of PPN either by ascending 
pathways from the DCs or by indirect descending 
pathways from the cortex might play a crucial role in 
the alleviation of PD symptoms (Fig. 3). 


The Spinal Cord as a Potential 
Common Neural Pathway for 
Treating Multiple Brain Disorders 
With Electrical Stimulation 


The spinal cord serves as a true “neural information 
highway” transmitting non-noxious as well as noci- 
ceptive sensory information from various parts of the 
body to supra-spinal structures in the brain. Although 
spinal cord stimulation has been used for the treat- 
ment of chronic pain for decades, its role as a channel 
to deliver therapeutic electrical signals to the cortex 
and other subcortical targets has never been 
explored.*! Traditionally, it has been assumed that, to 
cause cortical or subcortical neuronal modulation, the 
electrical stimulation has to be applied at the target 
site, which is evident in earlier approaches where 
intracortical microstimulation of areas 3b and 3a of 
the somatosensory cortex was used for creating artifi- 
cial sensation of flutter and proprioception, respec- 
tively.''7"!? However, more recently, thalamic 
stimulation or even peripheral nerve innervation have 
been used for the same purpose, suggesting that multi- 
ple locations along a neural pathway can be used for 
modulating the activity of the target of interest.!2%!*! 
Consistent with this argument, we had previously used 
electrical nerve stimulation to treat epileptic seizures.”* 
Recently we reported that closed-loop DCS can also 
be used to reduce the frequency and duration of seiz- 
ures in a Pentylenetetrazol (PTZ) rat model of epi- 
lepsy, primarily by modulating the theta frequency 
oscillations in the somatosensory cortex.'?” 

DCS has also been used for the remodeling of corti- 
cospinal projections in spinal cord injured rats, leading 
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to partial recovery of locomotion.'*? As mentioned 
previously, although functional magnetic resonance 
imaging studies revealed distinct modulation of corti- 
cal structures on the administration of DCS, it is also 
known to cause gene expression changes in upstream 
pathways.°”’! This suggests that the stimulation of 
the DCs does indeed result in the electrical signal 
being transmitted to multiple cortical and subcortical 
areas of great interest, from a therapeutic point of 
view, in the human brain. 

In addition, research on PD patients undergoing 
tango lessons has shown that the asymmetrical 
movements involved in this type of dancing could be 
providing the sensory feedback necessary for 
improvement in symptoms of balance and 
gait.'**!7° Because proprioceptive signals run_pri- 
marily through some of the largest myelinated axons 
that comprise the DC of the spinal cord, the obser- 
vations that increased proprioceptive feedback is 
correlated with improvements in PD symptoms 
strongly support our claim that high-frequency elec- 
trical stimulation of the dorsal funiculus of the spi- 
nal cord could provide therapeutic effects for PD. 
Accordingly, future implementation of a closed-loop 
DCS, which allows intermittent and not only contin- 
uous DCS, may provide a better strategy to maxi- 
mize this beneficial effect. 


Pathological Brain States as 
Neuronal Timing Disorders 


Synchronization of neuronal oscillations is represen- 
tative of the temporally precise interactions necessary 
for neural communication to occur between multiple 
areas in a neural circuit.'2° High-frequency oscilla- 
tions are observed in local neuronal areas, whereas 
slow oscillations recruit large networks. The relation 
between neural circuit architecture and oscillatory pat- 
terns is responsible for the dynamic establishment of 
normal brain operations carried out across spatial and 
temporal scales.'*” Earlier work in human patients 
using EEG emphasized the role of neural synchrony in 
cognitive functions, such as attention dependent stimu- 
lus selection and working memory tasks that require 
integration of distributed neural activity.!**!*° 
Although EEG recordings showed that synchroniza- 
tion of rhinal and hippocampal oscillations takes place 
during memory formation,'’® using multielectrode 
recordings we have demonstrated that hippocampal 
and prefrontal cortical oscillations synchronize during 
spatial-cognitive processes.'*! Ulhass and _ col- 
leagues'?”'? observed impaired neural synchrony in 
schizophrenia patients performing a cognitive task 
particularly reduced phase synchrony in the beta band 
(20-30 Hz), suggesting the relation between impaired 
neuronal synchrony and cognitive deficits associated 
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with schizophrenia. Previous work in our laboratory 
has shown that, although cross-structural synchroniza- 
tion of oscillations in the limbic circuit plays an 
important role in the manifestation of anxiety-related 
behaviors in healthy animals, deficits in the neuronal 
oscillations can lead to the development of behaviors 
that are typically characterized as bipolar mania disor- 
der.'°* We show that the nucleus accumbens—ventral 
tegmental area synchrony and nucleus accumbens— 
amygdala synchrony increases during specific phases 
of the elevated zero-maze task and that the correlation 
between synchrony and anxiety-related behaviors 
changes between healthy and diseased mice.'** 

EEG and functional magnetic resonance imaging 
studies of autistic children have also revealed reduced 
neural synchronization, characterized by a decreased 
functional connectivity between neural areas,'3>13° 
particularly in the cortical language system during 
comprehension of sentences'*” or between frontal and 
parietal areas during an executive task.'** Similarly, 
the reduction of alpha and beta band synchronization 
has been implicated in patients with Alzheimer’s dis- 
ease during the resting state,!'3”!4° and the reduction 
is correlated with the severity of the cognitive defi- 
cits.'41 Pijnenburg and colleagues!” showed that in 
Alzheimer’s disease patients, alpha and beta band syn- 
chronization decreased during a working memory 
task when compared with controls. Moreover, our 
research has demonstrated that neuronal synchroniza- 
tion increases between the amygdala and medial pre- 
frontal cortex circuits in a mouse model of 
depression.'*? Likewise, an increase in mesolimbic 
cross-structural coherence was correlated with hyper- 
activity and stereotyped behaviors in a model of 
hyponoradrenergia.'** 

These results suggest that many neurological disor- 
ders are characterized by pathological levels of neuro- 
nal synchronization, occurring in distinct neural 
circuits involved in key brain functions. In fact, in our 
decade-long survey of transgenic mice models of neu- 
rological/psychiatric disorders, we have always identi- 
fied a particular brain circuit in which pathological 
levels of neuronal oscillations or synchronization can 
be identified and found to correlate with the behavior 
abnormalities exhibited by each mouse strain. This led 
us to postulate that, independent of their specific etiol- 
ogy (eg, genetic, specific cellular degeneration, etc.), a 
large number of intrinsic brain disorders may produce 
abnormal motor, sensory, cognitive, and emotional 
outcomes as a result of fundamental disturbances in 
neuronal timing, taking place in specific brain circuits. 
Based on this view, we propose that any technique 
that can reset the pathological neuronal timing of a 
given neural circuit, bringing it close to a normal 
dynamic state, may be capable of ameliorating the 
symptoms, and hence the brain pathology, created by 
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such electrophysiological disturbance. Support for this 
contention came from the observation that neuronal 
activity recorded in both PD animal models and par- 
kinsonian patients often resembles the type of patho- 
logical neuronal hyper-synchronization that is 
prominent during epileptic seizures.'*°"'*° Our results 
with animal PD models and, more recently in an ani- 
mal model of epilepsy,'?* indicate that DCS may 
achieve its major therapeutic effects by disrupting 
ongoing supraspinal pathological synchronization by 
simultaneous activation of DC fibers in the spinal 
cord. 

Based on a decade of experimental work in animal 
models of brain disorders, we envision that the spinal 
cord “neural super highway” may become an optimal 
route to deliver different types of electrical, optical, 
and even noninvasive magnetic stimulation to manipu- 
late or even abolish pathological neuronal activity in 
supraspinal structures. This approach could trigger the 
establishment of a complete new repertoire of non- 
pharmacological, low-cost, and less invasive, neuro- 
physiologically inspired therapeutic options for 
treating a variety of neurological and even psychiatric 
diseases for which we have very few efficient treat- 
ments available today. 


Concluding Remarks 


After the original experimental findings in animal 
models of PD carried out in the late 2000s, multiple 
preliminary studies have reported the beneficial clini- 
cal effects of DC stimulation in PD patients. DCS 
effects are not only significant in motor symptoms 
such as akinesia, bradykinesia, and tremor, but they 
also provide significant alleviation of axial symptoms, 
gait and posture impairment, and “freezing,” a range 
of problems that are difficult to treat with current 
therapeutic options, such as DBS. Current experimen- 
tal evidence from our laboratory suggests that DCS 
modulates activity of subcortical and cortical struc- 
tures by activating ascending DC fibers of the spinal 
cord. Although the precise mechanism is not yet clear, 
high-frequency continuous DCS appears to disrupt the 
pathological low-frequency hypersynchronized neuro- 
nal firing observed in the basal ganglia and motor cor- 
tex, which has been associated with PD. Indeed, the 
neurophysiological effects observed with DCS in 
experimental animals mimic those observed during 
dopamine-replacement therapy. In addition, in 
rodents, DCS also has produced a long-term protective 
effect on the degeneration of dopaminergic neurons, 
making it a promising treatment option for early-stage 
PD along with traditional levodopa therapy. Because 
of its semi-invasive nature, we propose that in the 
future DCS may become a potential therapeutic alter- 
native for DBS. e 
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BRAIN-MACHINE INTERFACES: FROM BASIC 
SCIENCE TO NEUROPROSTHESES AND 
NEUROREHASBILITATION 


Mikhail A. Lebedev and Miguel A. L. Nicolelis 


Duke University, Durham, North Carolina 


Lebedev MA, Nicolelis MAL. Brain-Machine Interfaces: From Basic Science to 
Neuroprostheses and Neurorehabilitation. Physiol Rev 97: 767-837, 2017. Pub- 
lished March 8, 2017; doi:10.1152/physrev.00027.2016.—Brain-machine inter- 
“i @ faces (BMls) combine methods, approaches, and concepts derived from neurophysiology, 
- Wi computer science, and engineering in an effort to establish real-time bidirectional links 
between living brains and artificial actuators. Although theoretical propositions and some proof of 
concept experiments on directly linking the brains with machines date back to the early 1960s, BMI 
research only took off in earnest at the end of the 1990s, when this approach became intimately 
linked to new neurophysiological methods for sampling large-scale brain activity. The classic goals 
of BMls are 7) to unveil and utilize principles of operation and plastic properties of the distributed 
and dynamic circuits of the brain and 2) to create new therapies to restore mobility and sensations 
to severely disabled patients. Over the past decade, a wide range of BMI applications have 
emerged, which considerably expanded these original goals. BMI studies have shown neural control 
over the movements of robotic and virtual actuators that enact both upper and lower limb 
functions. Furthermore, BMls have also incorporated ways to deliver sensory feedback, generated 
from external actuators, back to the brain. BMI research has been at the forefront of many 
neurophysiological discoveries, including the demonstration that, through continuous use, artificial 
tools can be assimilated by the primate brain’s body schema. Work on BMlls has also led to the 
introduction of novel neurorehabilitation strategies. As a result of these efforts, long-term contin- 
uous BMI use has been recently implicated with the induction of partial neurological recovery in 
spinal cord injury patients. 
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I. INTRODUCTION 767 the true holy grail of neuroscience since the first images of 
Il. HISTORY OF BMI RESEARCH 772 brain circuits were produced, more than 100 years ago, by 
I. BMI CLASSIFICATION 779 the skillful hands of Santiago Ramon y Cajal (256, 658). By 
IV. REPRESENTATION OF INFORMATION... 782 the time Cajal’s histological approach was complemented 
V. MULTICHANNEL RECORDING... 786 by its electrophysiological counterpart, Sir Edgar Adrian’s 
Vi. DECODING OF BRAIN SIGNALS 792 metal microelectrode approach to record the electrical 
Vil. MOTOR CONTROL WITH... 795 pulses produced by individual neurons (2), the neuron doc- 
Vill. NONINVASIVE BMis 804 trine (732) had become established as the fundamental the- 
IX. BMIs WITH ARTIFICIAL SENSATIONS 806 ory in the emergent field of brain research. This doctrine 
xX. COGNITIVE BMIs 810 purports that individual neurons work as the functional 
XI. |BRAIN-TO-BRAIN INTERFACES... 811 unit of the brain through processing and transmission of 
XII. | BMIAS A POTENTIAL... 814 electrophysiological signals. 

Xill. CONCLUSION 817 


Despite the undeniable success of the neuron doctrine, since 
the origins of modern neuroscience, researchers entertained 
physiological models of brain function in which popula- 
tions of neurons performed the fundamental job of gener- 
ating functions and behaviors. Indeed, the Italian anato- 
mist, Camillo Golgi, who shared the 1906 Nobel Prize in 
Medicine and Physiology with Ramon y Cajal, was the first 


I. INTRODUCTION 


A. From a Single Neuron to Neural 
Ensembles 


The daunting task of unraveling the physiological mecha- 
nisms that account for the operation of the human brain, a 
highly complex and self-adaptive biological system, formed 
by ~100 billion interconnected neurons (37), has become 
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to introduce the term neural network, as a way to describe 
the underlying “functional module” of brain operation pro- 
posed in his reticular theory (308). According to Golgi’s 
view, brain tissue should work pretty much like the heart 
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myocardium, forming a syncytium, or a continuous net- 
work of fused or tightly connected neurons (308). Cajal’s 
histological demonstration of the existence of synaptic 
clefts between neurons in most of the brain debunked Gol- 
gi’s view from an anatomical point of view and, as a con- 
sequence, the reticular theory was abandoned and the term 
network fell out of favor with most neuroscientists. Ironi- 
cally, many decades later, clusters of neurons tightly linked 
via gap junctions (155) were identified in some key struc- 
tures of the mammalian brain, including the inferior olive 
(506), hippocampus (443), and neocortex (486, 747). Be- 
cause of the abundant existence of tight junctions, these 
clusters exchange information through electrotonic cou- 
pling. As such, they clearly typify the type of networks 
envisioned by Golgi. 


During the century of intense work that followed the pio- 
neering discoveries of Ramon y Cajal and Golgi, many 
other histological, electrophysiological, and imaging methods 
have been incorporated in the technical arsenal employed by 
neuroscientists to probe brain function at different levels of 
spatial and temporal resolution (FIGURE 1,A-C ). For most of 
this continuous procession of new technological develop- 
ments, the neuron doctrine continued to flourish. As a result, 
for the vast majority of the neuroscientific community, the 
single neuron remained the central focus of systems neurosci- 
ence for most of the 20th century (583). 


Yet, since the late 1940s, a neural network-based view of 
brain function began to reemerge. Inspired by the pioneer- 
ing work of Thomas Young on color coding in the early 
19th century (881) and that of Charles Sherrington on spi- 
nal reflexes at the beginning of the 20th (733), theoreticians, 
such as Donald Hebb (352), and neurophysiologists, such 
as John Lilly (499), proposed that the true functional unit of 
complex brains, such as ours and those of other mammals, 
is represented, according to Hebb’s own terms, by “... a 
diffuse structure comprising cells in the cortex and dien- 
cephalon, capable of acting briefly as a closed system, de- 
livering facilitation to other such systems (352).” 


Even though Hebb’s masterpiece work, The Organization 
of Behavior (352), published in 1949, launched the modern 
era of neural population coding in systems neuroscience, 
few took notice at the time of its publication. By then, 
neurophysiologists were primarily engaged in characteriz- 
ing the physiological properties of individual neurons, ei- 
ther using the classical Adrian’s approach of extracellular 
single neuron recording (2, 372) or a new methodological 
breakthrough of that time: intracellular single neuron re- 
cordings using sharp glass electrodes (92, 503, 781). Cer- 
tainly, the multiple technological challenges involved in de- 
veloping techniques for recording simultaneously from 
large populations of individual brain cells, even in anesthe- 
tized animals, let alone awake preparations, kept most ex- 
perimentalists away from trying to test experimentally the 
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FIGURE 1. Temporal and spatial resolution for different tech- 
niques to study the brain and interact with its circuitry. A: techniques 
for recording and visualizing. B: electrode techniques for recording 
and stimulation. C: techniques for stimulation and making lesions. 
(Adapted with permission from Sejnowski TJ, Churchland PS, 
Movshon JA. Putting big data to good use in neuroscience. Nature 
Neurosci 17: 1440-1441, 2014. Reprinted by permission from 
Macmillan Publishers Ltd.) 
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ideas about neural populations introduced by Hebb. Thus, 
if one combines these notorious and, at the time, insur- 
mountable technical difficulties, with the stupendous suc- 
cess and widespread acceptance of Cajal’s neuron doctrine, 
amplified significantly in the 1960s and 1970s by the enor- 
mous impact of the Hubel and Wiesel’s characterization of 
single neuron physiological properties in the visual cortex, 
it is no surprise that a neural population view of brain 
function had to wait for four long decades before it could 
begin receiving serious experimental attention by neuro- 
physiologists. 


Among the experimental studies that rekindled the interest 
in neural populations in neurophysiology during the 1980s 
was the pioneering work of Apostolos Georgopoulos on 
directional coding in the primate motor cortex (292, 295, 
299) and John O’Keefe’s discovery of place neurons in the 
rat hippocampus (600). Georgopoulos’ findings that indi- 
vidual neurons in the primate primary motor cortex are 
broadly tuned to the direction of arm movement (294, 295, 
423, 710) and that populations of such neurons, rather than 
an individual M1 neuron, have to be pooled together to 
compute the direction in which a monkey is about to move 
its arm (292, 299) brought to the forefront the much ne- 
glected, if not forgotten, Hebbian’s view of the neural pop- 
ulation basis of brain function. By the mid-1990s, the in- 
troduction of new electrophysiological methods for chronic 
multielectrode recordings in freely behaving animals trig- 
gered a new phase of neural ensemble physiology (581, 582, 
584, 857, 858). In this experimental approach, arrays or 
bundles of microelectrodes, originally made of fine insu- 
lated metal filaments, were chronically implanted across 
multiple cortical and subcortical structures of the brain of 
rodents and primates and yielded viable single and multi- 
unit activity for long periods of time, which today can reach 
several years for the same animal (711) (FIGURE 2). By the 
early 1990s, this new approach was yielding simultaneous 
recordings of ~12-24 neurons in freely behaving rats. 
These recordings lasted for several weeks or even months 
(578, 579, 584). By mid-1995 the yield increased to ~50 
simultaneously recorded neurons, with the added capability 
that individual neurons could be recorded in up to five 
different subcortical and cortical structures that defined a 
given neural pathway (i.e., the rat trigeminal somatosen- 
sory system) (578). By 1999-2000, this recording bench- 
mark reached 100 neurons, and such simultaneous record- 
ings could be obtained in both awake rats and monkeys 
(580-582). At that time the modern concept of BMIs was 
proposed by John Chapin’s and Miguel Nicolelis’ laborato- 
ries working together (124, 575, 852). 


B. Emergence of BMIs 


The modern era of BMIs emerged precisely at the time the 
methods for chronic multi-electrode recordings were con- 
solidated in rodents and started to make their transition to 
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FIGURE 2. Multichannel, wireless recordings in rhesus monkeys. 
A: movable 10 x 10 microwire arrays. B: photographs showing 
implant connectors in two monkeys (/eft and middle) and wireless 
module (right). C: layered schematic of the 3-D printed modular 
headcap. D: headcap wireless assembly. [Adapted from Schwarz et 
al. (711).] 


primates, first to New World monkeys (582, 852) and then 
to rhesus monkeys (114, 463, 580). Around that time, the 
term BMI was introduced for the first time in the systems 
neuroscience literature (575). The main goal of these studies 
was to investigate physiological properties, including the 
ability of neural ensembles in the sensorimotor cortex to 
encode information and express plastic adaptations while 
freely behaving animals learned new motor tasks (460). 
However, with the publication of a series of original stud- 
ies, conducted in rats and monkeys in the early 2000s, it 
soon became apparent that BMIs could also serve as the 
foundation of a new generation of neuroprosthetic devices 
aimed at restoring mobility to patients severely paralyzed 
due to trauma to the nervous systems, notably spinal cord 
injuries (SCIs) or neurodegenerative diseases. FIGURE 3 
shows the original schematic description of this idea, pro- 
posed in the early 2000s, as an envisioned direct link be- 
tween a human brain and a robotic arm. FIGURES 4 AND 5 
illustrate how the original BMI control scheme was adapted 
to become the experimental paradigm employed with rhe- 
sus monkeys. For the most part, until today, most labora- 
tories around the world that work with upper limb BMIs 
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continue to employ the elements depicted in FIGURES 3-5, 
which were originally published about 17 years ago (124, 
852). Indeed, most researchers working in the field would 
consider FIGURE 3 as the standard or operational definition 
of a BMI, which can be described as a reciprocal link be- 
tween an animal or human brain and an artificial actuator, 
such as a virtual or robotic arm or leg, which allows the 


Brain—machine interface 


Computer control 


FIGURE 4. A brain-machine interface for enabling arm move- 
ments and with multiple feedback loops. A rhesus monkey is con- 
trolling a robotic arm that reaches and grasps objects. The robotic 
arm contains touch and position whose signals are processes and 
converts to microstimulation pulses delivered to the sensory areas 
in the brain. Neuronal ensemble activity is recorded in multiple brain 
areas and decoded to generate commands that control the robotic 
arm. [From Lebedev and Nicolelis (466), with permission from 
Elsevier. ] 
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FIGURE 3. Schematics of a cortical brain-ma- 
chine interface. Intracranial recordings are em- 
ployed to sample the extracellular activity of sev- 
eral hundred neurons in multiple cortical areas 
that are involved in motor control of arm and 
movements. The combined activity of cortical neu- 
ronal ensembles is processed, in real time, by a 
series of decoders that extract motor parameters 
from the brain signals. The outputs of these de- 
coders are used to control the movements of a 
robot arm that allows human patients to perform 
arm movements. [From Nicolelis (575).] 


subject to utilize its own volitional electrical brain activity 
to control the movements of such an actuator, while receiv- 
ing continuous feedback information from it. 


As we will see throughout this review, many aspects of this 
standard definition have been expanded significantly over 
the past two decades. For example, as illustrated in FIGURE 
5A, large variety of brain signals, ranging from single units 
(114, 124, 466, 583, 852), to local field potentials (LFPs) 
(264, 670, 750), electrocorticography (ECoG) (483, 484, 
697, 759, 837, 839, 856), electroencephalography (EEG) 
(7, 71, 141, 169, 358, 412, 627, 865), all the way to mag- 
netic resonance imaging (MRI) signals (666, 745, 755, 847, 
879), have been utilized as the source of voluntary motor 
activity needed to control an artificial actuator. By the same 
token, BMIs have been now designed to control a large 
variety of actuators, including computer cursors (114, 283, 
463, 692, 725, 794, 864), digital communication systems 
(10, 71, 101, 247, 545, 559, 672), robotic limbs (114, 152, 
360, 463, 795, 817), robotic exoskeletons (156, 456, 467, 
836), avatar bodies (151, 377, 435, 598, 657, 835), drones 
(426, 457), and wheelchairs (169, 511, 553, 656, 890). 
BMIs have also been coupled with a variety of traditional 
medical devices (522). For instance, recently a BMI core 
was used to allow patients to control stimulators that acti- 
vate their own muscles through functional electrical stimu- 
lation (FES) (83, 633) (FIGURE 6). 


To extract the type of information needed to control such a 
vast list of actuators, a multitude of mathematical and com- 
putational approaches have been proposed as potential 
real-time decoders of voluntary motor activity generated by 
the brains of animals and human subjects (49, 449, 490). 
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A 
Multichannel Neural 
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FIGURE 5. The first brain-machine interface for reaching and grasping. A: diagram of the experimental 
setup. The extracellular electrical activity of neuronal ensembles was simultaneously recorded from multiple 
cortical areas and then directed to a robotic arm that performed reaching and grasping movements. B: 
schematics of three motor tasks. In task 1, monkeys placed the cursor over a visual target. In task 2, the 
joystick was in a fixed position, and the monkeys grasped a virtual object by squeezing the joystick handle. In task 
3, monkey placed the cursor over the target then produced a gripping force. C: recordings of motor 
parameters (blue lines) and their decoding (red lines). From top to bottom: example traces of hand velocity (Vx, 
Vy) and gripping force (GF). [Adapted from Carmena et al. (114).] 


Moreover, a variety of signals and strategies for delivering 
sensory feedback to subjects operating a BMI have been 
proposed, including visual feedback through computer 
screens (114, 377, 692), tactile and proprioceptive cues via 
haptic displays (128, 597), and even direct intracortical 
microstimulation (ICMS) applied to the primary somato- 
sensory cortex (S1) (262, 597, 598). 


To give an idea of how energized the field has become, even 
BMIs that involve the collaboration of multiple animal 
brains have been demonstrated in rat and monkeys (614, 
616, 657). Notwithstanding these countless innovations, 
FIGURES 3 AND 4 still are useful to describe the basic ele- 
ments that constitute a BMI. These include the use of sen- 
sors (e.g., multielectrode arrays) to sample large-scale brain 
activity; multi-channel electronics that amplify, filter, and 
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digitize these signals; a computational engine responsible 
for real-time extraction of motor commands from the raw 
brain activity; an artificial actuator that performs motor 
tasks; and a stream of feedback signals delivered to the 
subject’s brain. 


Merely 17 years after the modern age of BMI research was 
launched, the tremendous impact of this paradigm in neu- 
roscience can be measured by a variety of metrics. For ex- 
ample, a bibliographic search in Google Scholar using the 
terms brain-machine interface and a closely related expres- 
sion, brain-computer interface, yields more than 40,000 
publications during the past decade alone. Meanwhile, sev- 
eral of the key experimental and clinical papers in the field 
have surpassed 1,000 citations. Several prominent books 
have been published on different aspects of BMIs (61, 106, 
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FIGURE 6. Hand grasping induced in a tetraplegic patient by func- 
tional electrical stimulation (FES) controlled by an EEG-based brain- 
machine interface (BMI). The BMI utilized motor imagery of foot 
movements. FES was applied through surface electrodes. Aided by 
this BMI, the patient was able to grasp a glass with his paralyzed 
hand. 


210, 577, 586, 660, 862). During the same period, an esti- 
mated $800 million United States dollars have been in- 
vested in BMI research worldwide. 


Since the introduction of experimental BMIs in the late 
1990s, many applications have emerged for healthy sub- 
jects, outside the domains of basic and clinical research. 
BMIs for computer gaming (6, 9, 257, 342, 489, 493, 531, 
532, 775, 874), EEG-based that detect drowsiness in drivers 
(137, 286, 501, 504, 636, 685, 810), and even BMIs for 
education (371, 526) are just a few examples of this parallel 
line of BMI development outside biomedical research. As a 
result, many started to consider BMIs as a method to aug- 
ment human neural and physiological functions, such as 
cognitive abilities (242, 518, 519, 539, 888) and motor 
performance (148). Despite being very interesting subjects 
for debate, none of these latter areas of BMI application will 
be covered here. 


In this review, we start with a brief history of BMIs, fol- 
lowed by a description of major classes of BMIs. Next, we 
spell out the major components for building a BMI: sensors 
to record large-scale brain activity, decoding algorithms for 


J/2 


extracting behavioral variables from brain signals, the 
means to deliver sensory feedback to the brain, and external 
actuators controlled through BMIs. Finally, we will discuss 
BMI applications to restore mobility and, eventually trigger 
partial neurological recovery in severely paralyzed patients. 


ll. HISTORY OF BMI RESEARCH 


A. Early Studies 


As mentioned in the introduction, the history of BMIs is 
intimately related to the effort of developing new electro- 
physiological methods to record the extracellular electrical 
activity of large neuronal populations using multi-electrode 
configurations. Such an essential component of modern 
BMI architecture was pioneered in the 1950s, by John Cun- 
ningham Lilly, then a principal investigator at the National 
Institutes of Health. In his experiments, Lilly was able to 
implant 25-610 electrodes, either on the pial surface of the 
cortex or intracortically, in adult rhesus monkeys (498) 
(FIGURE 7). In addition to recording field potentials (25 
channels at a time) with these electrodes while animals ex- 
hibited a variety of behaviors and states (arm movements, 
sleep, etc.), Lilly also applied electrical current through 
those electrodes and elicited movements in both anesthe- 
tized and awake monkeys (496). He observed that motor 
responses could be evoked from many cortical sites, includ- 
ing M1 and S1. Lilly concluded that there was no clear-cut 
separation between cortical regions presumed to be motor 
alone or sensory alone. He suggested that these areas be 
named sensorimotor instead (497). 


The next intermediary step in the development of the BMI 
concept can be traced back to the introduction of “EEG 
biofeedback” or “neurofeedback,” which became very 
popular in the 1960s and 1970s, in a variety of experimen- 
tal settings (179, 258, 405, 407, 622, 749, 764, 777, 843, 
867). In these studies, subjects were provided with an indi- 
cator of their own neural activity, for example, auditory or 
visual feedback derived from EEG recordings, which as- 
sisted them with self-regulating those neural signals. David 
Nowlis and Joe Kamiya (405, 595, 596) recorded EEGs in 
animals and human subjects, and converted them into 
sound. Aided by this type of neurofeedback, subjects gained 
some level of volitional control over their own EEG activity. 
Maurice Sterman and his colleagues converted EEGs of ep- 
ileptic patients into lights and tones, and achieved seizure 
reduction with this type of neurofeedback training (763- 
766). 


According to Daniel Dennett (193), the first experiment in 
which human subjects sent brain-derived signals to com- 
mand an external device was described in 1963 by Grey 
Walter in a talk to the Osler Society at Oxford University. 
Since Walter himself did not publish this presentation, Den- 
nett’s anecdote may not be completely accurate or verified. 
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BRAIN-MACHINE INTERFACES 


Published literature, however, confirms that Walter and his 
colleagues implanted multiple electrodes (up to 1,000) in 
the cortex of neurological patients and used these implants 
for monitoring cortical field potentials over a period of 
several months (831). In the experiments described by Den- 
nett, Walter recorded motor cortex readiness potentials 
preceding movements, also called Bereitschaftspotentials, 
while his patients periodically pressed a button to advance 
slides in a slide projector. The button presses were self- 
paced. The readiness potentials led the movement by ap- 
proximately half a second and were sufficiently strong to be 
detected by Walter’s recording equipment. Next, Walter 
succeeded in creating a direct link between each patient’s 
motor cortex and the projector. In this experimental condi- 
tion that we would now call brain control, the button was 
electrically disconnected from the projector and the slides 
were advanced by motor cortical readiness potentials. It 
came as a surprise to the patients that the projector re- 
sponded to their will even before they physically initiated 
the movement. Although Walter’s experiments can be con- 
sidered as the first proof of concept of the possibility of 
building BMIs, he never published these results or inter- 
preted them in the context of BMIs, even though in his 
earlier career in the 1950s, he conducted research on robots 
with artificial brains (829, 830). 


By the end of the 1960s, researchers at the NIH Labora- 
tory of Neural Control started to experiment with the 
possibility of utilizing recordings from cortical neurons 
to control artificial actuators (270). They were also in- 
terested in using direct connections between brains and 
external devices to restore hearing to the deaf, walking to 
the paralyzed, and vision to the blind (701). This NIH- 
led research was conducted with some universities and 
medical schools participating as subcontractors. Karl 
Frank, the NIH laboratory head, proclaimed, “We will 
be engaged in the development of principles and tech- 
niques by which information from the nervous system 
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FIGURE 7. Multichannel implant of John 
Lilly. A: parts of the electrode implant: the 
lower end of a spear-shaped hardened 
steel tool that was used for starting bone 
holes (a, a’); the lower part of the mandrel 
(b, b’) with a sleeve on the cylindrical lower 
end (b’); sleeves (c, c’); electrode (d, d’)); 
and sleeve guide (e). B: an X-ray of a mon- 
key skull showing 20 implanted sleeves and 
one inserted electrode. [Adapted from Lilly 
(498). Reprinted with permission from 
AAAS. ] 


can be used to control external devices such as prosthetic 
devices, communications equipment, teleoperators .. . and ul- 
timately perhaps even computers” (269). 


In their initial study, the NIH team implanted five micro- 
electrodes in the primary motor cortex (M1) of rhesus mon- 
keys and then recorded action potentials generated by 3-8 
M1 neurons, while animals performed a motor task that 
required them to flex and extend their wrists (374). Since 
the eventual goal was to convert these neuronal signals into 
the movements of an external device, the researchers 
probed whether wrist movements could be predicted from 
the recorded activity of small neuronal populations. They 
utilized multiple linear regression as a prediction algorithm. 
The algorithm took neuronal rates as inputs and returned 
movement kinematics as the output. A decade of this re- 
search eventually resulted in the demonstration of a real- 
time neural control (702): a rhesus monkey with 12 micro- 
electrodes implanted in M1 for 37 mo learned to move a 
cursor on an LED display using its own neural activity as a 
direct source of motor commands. 


During the late 1960s, Eberhard Fetz and his colleagues 
conducted experiments in which they utilized the electrical 
activity of single neurons recorded in monkey M1 as a 
source of neurofeedback (254). Using this apparatus, mon- 
keys learned to self-regulate their own single-neuron activ- 
ity. Typically, one neuron was tested at a time. The neuro- 
nal electrical firing rate was converted into either auditory 
(a click for each spike) or visual (deflections of an arrow 
meter placed in front of the animal) feedback. Monkeys 
learned to volitionally modulate the activity of each indi- 
vidual M1 neuron to reach a particular level of firing re- 
quired to obtain a reward (FIGURE 8). While Fetz empha- 
sized the neurofeedback aspect of such operant condition- 
ing experiments, Brindley and Craggs (89, 168) employed 
epidural recordings of motor cortical field potentials in the 
frequency band 80 to 250 Hz in baboons to test the possi- 
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bility of creating a motor neuroprosthesis that recognized 
specific movements produced with the arms or legs. More- 
over, Craggs (167) used baboons with complete spinal cord 
transections at the midthoracic level as a model of human 
paraplegia, while he recorded motor commands directly 
from the cortical representation of the foot disconnected 
from its spinal cord projection area. 


At the same time these laboratories experimented with ex- 
traction of motor signals from the brain and/or using them 
to generate neurofeedback, another line of research focused 
on ways to deliver information to the brain using electrical 
stimulation, either applied to peripheral nerves (153, 354) 
or to the central nervous system (90, 91, 495). This work 
led to early attempts to build sensory BMIs that strived to 
restore normal perception to patients suffering with neuro- 
logical conditions that induced significant sensory deficits. 
From these pioneering studies, the work on cochlear im- 
plants (FIGURE 9) eventually reached the most spectacular 
results (198, 218, 219, 367, 510, 743, 855). In parallel, 
some progress was achieved in the development of visual 
cortical prostheses pioneered by the groups led by Brindley 
(88, 90, 91) and Dobelle (200-203). These researchers ap- 
plied electrical current to the visual cortex of blind patients 
through grids of surface electrodes. Using this apparatus, 
blind subjects could perceive light spots, phosphens, and 
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35 


Reward 


FIGURE 8. Operant conditioning of the dis- 
charge rate of a single motor cortex neuron. 
A: a monkey was operantly conditioned to in- 
crease the neuron’s rate until it reached a 
target level, and a reward was delivered. The 
feedback of the neuron’s firing was provided 
using either a visual meter or auditory clicks, 
one slick per spike. B: after the training, the 
monkey could increase the neuronal rate even 
in the absence of the feedback. However, re- 
moval of reinforcement extinguished the neu- 
ronal rate increase. [Adapted from Fetz 
(254). Reprinted with permission from 
AAAS. ] 


Pellets and clicks 


40 45 


learned to recognize simple visual objects composed of sev- 
eral phosphens. 


Also in the 1960s, Bach-y-Rita and his colleagues started to 
develop visual substitution systems for the blind, based on 
tactile stimulation of the skin on the patient’s back (39, 40). 
This technique became known as vision substitution by tac- 
tile image projection. The apparatus employed, called a 
haptic display, consisted of 400 solenoid stimulators ar- 
ranged in a 20 by 20 array. The tactile stimulation was 
applied to the surface of the patient’s back and attempted to 
reproduce, through the sense of touch, visual images cap- 
tured by a video camera (FIGURE 10). After being trained for 
10 h, blind patients learned to recognize objects and their 
positional relationship in a room, as well as landmarks, 
such as the room’s door frame (40). 


B. Explosive Development in the Late 1990s 


Despite the initial push observed in the 1960s and 1970s, 
research on direct links between brain and machines expe- 
rienced a decline during the next 20 years. Clearly, the lack 
of major technological breakthroughs in the area of multi- 
channel neuronal recordings during that period prevented 
the field from taking off and fulfilling the auspicious poten- 
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tial envisioned by the early proponents of this research pro- 
gram. 


Around the mid-1990s, however, the required technologi- 
cal innovation materialized with the introduction of a new 
multi-electrode design that allowed bundles or planar ar- 
rays to be built using flexible and insulated metal filaments, 
known as microwires (108, 578, 584, 585). John K. Cha- 
pin, one of the pioneers of microwire implants, discovered 
that if a blunt tip was left exposed in an otherwise insulated 
metal microwire, made of stainless steel, individual single- 
units could be recorded for many weeks or even months 
from the forepaw representation area of M1 and S1 of 
freely moving rats (125, 734). A few years later, Miguel 
Nicolelis and Chapin chronically implanted multiple bun- 
dles and/or arrays of Teflon-coated stainless steel mi- 
crowires in the ventral posterior medial nucleus (VPM) of 
the thalamus of adult rats and recorded the simultaneous 
electrical signals produced by up to 24 of these thalamic 
neurons in awake and freely moving rats (579, 584). A year 
later, the same authors reported simultaneous multi-site 
recordings, obtained from chronic implants that included 
not only VPM, but also multiple key subcortical structures 
that define the rat trigeminal system, such as S1, different 
thalamic and brain stem nuclei and even the trigeminal 
ganglion of the same subjects (578). Using this new ap- 
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FIGURE 9. Diagram of a cochlear im- 
plant. (Reprinted with permission from 
MED-EL Medical Electronics GmbH, Inns- 
bruck, Austria.) 
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proach, these researchers recorded the extracellular activity 
of up to 48 neurons distributed across multiple subcortical 
and cortical relays of the rat trigeminal somatosensory sys- 
tem, in awake, freely behaving rats. These experiments 
marked the first time simultaneous neuronal population 
activity, originating from multiple processing levels of a 
mammalian sensory system, was measured in the same an- 
imal subject. Three years later the same technique was val- 
idated in primates, allowing the Nicolelis Lab at Duke Uni- 
versity to record from multiple cortical areas in awake owl 
monkeys (582). 


Since the recording properties of chronic microwire im- 
plants lasted for several months in both rats and owl mon- 
keys, Chapin and Nicolelis found this technique suitable for 
testing the concept of linking the brains of rats and monkeys 
to artificial actuators and investigating whether these ani- 
mals could learn to control external devices using only their 
brain electrical activity. In 1999, Chapin and Nicolelis pub- 
lished their first BMI study where rats learned to use the 
combined extracellular activity of up to 46 neurons to con- 
trol the uni-dimensional movements of a lever that deliv- 
ered water, collected from a water dropper, to the animal’s 
mouth (124). Initially, rats were trained to press a bar to 
generate the lever movements. As rats learned this task, a 
principal component analysis algorithm, implemented us- 
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FIGURE 10. Vision substitution system developed by Paul Bach-y- 
Rita and his colleagues. The system included a digitally sampled 
television camera, control electronics, and a 400-point matrix array 
of tactile stimulators mounted on a dental chair. The tactile stimu- 
lation projected images to the back of blind subjects. [From Bach-y- 
Rita et al. (40).Reprinted by permission from Macmillan Publishers 
Ltd.] 


ing analog electronics, was employed to transform the com- 
bined cortical activity of the recorded neuronal popula- 
tion into a continuous motor control signal that moved 
the lever. To the total surprise of Chapin and Nicolelis, 
not only did the rats learn to operate the lever efficiently 
to drink their daily water allotment, but, in a few trials, 
the animals would simply stop moving their forepaws 
and still successfully use their brain activity alone to 
move the lever and receive their water reward. A year 
later, the Nicolelis laboratory demonstrated that owl 
monkeys could utilize the simultaneously recorded elec- 
trical activity of close to 100 cortical neurons, distributed 
across multiple frontal and parietal cortical fields, to con- 
trol the two- and three-dimensional movements of a mul- 
tiple degree of freedom robot arm (852). This study also 
introduced a new analytical method and graphic repre- 
sentation, the neuronal dropping curve (NDC), which 
would become a standard representation to depict the 
neurophysiological results of BMI studies (FIGURE 11) 
(114, 283, 377, 462, 465, 562, 690, 692). In 2000, ina 
review paper commissioned by Nature, Nicolelis dubbed 
the paradigm Chapin and he had implemented as a brain- 
machine interface or BMI, the first time the now tradi- 
tional term was used to refer to real-time links between 
living brains and artificial devices (575). 


776 


Curiously, also in 1999, without knowledge of the work 
carried out by Chapin and Nicolelis in animals, a group led 
by Niels Birbaumer at the University of Tubingen in Ger- 
many pioneered their version of a direct link between a 
brain and a computer in locked-in patients (71). Birbaumer 
chose to name this paradigm brain-computer interface 
(BCI), a term that had been introduced in the literature by 
Jacques Vidal in 1973 (823). Birbaumer’s BCI allowed 
locked-in patients to communicate with the external world 
using slow cortical potentials recorded via a noninvasive 
technique, EEG recordings, to control computer software 
for spelling. Using this system, locked-in patients became 
capable of writing messages on the computer. 


Overall, the original papers by Chapin, Nicolelis, and 
Birbaumer mark the beginning of the modern age of re- 
search on BMIs. In the case of intracranial BMIs, the focus 
of this review, advances in multi-electrode recording meth- 
ods, combined with the introduction of faster digital com- 
puters running new computational algorithms for extract- 
ing motor signals from brain-derived signals, triggered a 
phase of very fast growth in the field. Thus, following the 
original demonstrations in rats and New World monkeys, 
the next important milestone of the field was the translation 
of the BMI paradigm to rhesus monkeys, the conventional 
experimental animal model for exploring neurophysiology 
of advanced motor behaviors and cognition. In a quick 
succession, three different groups published their results in 
this primate species (114, 725, 794). In a span of 12 mo, the 
BMI paradigm in rhesus monkeys incorporated the use of a 
series of novel actuators, such as a computer cursor (725, 
794), and a robot arm capable of producing both arm 
reaching movements and hand grasping (114). With this 
latter addition, the Nicolelis laboratory demonstrated that 
the same pool of recorded cortical neurons could be em- 
ployed to simultaneously extract hand gripping force and 
arm position and velocity from multiple frontal and parietal 
cortical areas in awake rhesus monkeys (114, 463). 


In 2004, the Nicolelis laboratory also reported the first 
demonstration that ensembles of subcortical neurons, re- 
corded intraoperatively with microwire bundles, could be 
employed to extract hand movements in awake and con- 
scious human subjects (620). These recordings were ob- 
tained during a neurosurgical procedure in which Parkinso- 
nian patients received a deep brain stimulator. During a 
brief intraoperative period of 10-15 min, up to 50 neurons 
located in the subthalamic nucleus and thalamic motor nu- 
clei were recorded while these patients played a one-dimen- 
sional video game by exerting a gripping force with one 
hand. This study revealed that the same computational al- 
gorithm, multi-linear regression, employed by Wessberg et 
al. (852) and Carmena et al. (114) in owl and rhesus mon- 
keys, respectively, could be used to extract hand movement 
patterns from human subcortical signals. 
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A couple of years later, in 2006, the group led by John 
Donoghue reported the operation of a BMI in one patient 
chronically implanted with a different multichannel record- 
ing technology approved for human trials (362). Their im- 
plant, named the Utah probe, was a 10 X 10 array of sili- 
con-etched rigid needles (111, 537, 593). The Utah probes 
were inserted in the cortex ballistically using a pneumatic 
gun (680), the method adopted for this implant to avoid the 
“bed of nails effect,” where a slowly inserted dense elec- 
trode array produces cortical dimpling and trauma (76, 
111, 668). A total of two patients were implanted with this 
device in 2006 (362), one of whom experienced implant 
malfunctions for the first 6 mo followed by 2 mo of record- 
ings. Both patients used a BMI to control two-dimensional 
movements of a computer cursor. 


As discussed below, recordings with Utah array suffer from 
biocompatibility issues (220, 251, 679), which in our opin- 
ion should preclude them for further use in human subjects. 
Shortly after implantation, this probe can produce a signif- 
icant tissue lesion and, hence, become encapsulated by glia 
and protein deposits, as a result of the local inflammatory 
reaction. Usually, this process renders the Utah probe un- 
usable for single-unit recordings after a few weeks/months 
(134). Groups that rely on this probe usually resort to the 
utilization of a threshold-crossing method to detect useful 
neuronal activity (134, 803), a maneuver that discards well- 
established neuronal recording quality criteria (580) and 
increases the likelihood of recording noise instead of neu- 
ronal spikes, by mistakenly recording mechanical, electrical 
and EMG artifacts as if they represented valid neural activ- 
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ity. As such, this recording technology cannot be considered 
as the final solution for clinical BMI applications, even 
though currently this method is commonly used in human 
trials. Practical solutions will clearly require better record- 
ing reliability, stability, and longevity standards to become 
accepted by patients and clinicians. 


To achieve better biocompatibility of a brain implant, in 
1989, Philip Kennedy implanted an ALS patient with a 
neurotrophic electrode loaded with nerve growth factors 
that induced growth of nerve fibers into the electrode tip 
(418, 420, 422). The study reported that the patient learned 
to produce on/off neural control signals that were detected 
by the electrode. While the same group continues this re- 
search until now (98, 327), neurotrophic electrodes were 
not adopted by other groups and their effectiveness remains 
difficult to evaluate. 


In addition to a few more clinical demonstrations of 
BMIs that controlled computer cursors (3, 731), upper 
limb robotic prostheses (152, 360) or an FES system for 
the hand (83) using either populations of M1 or other 
cortical neurons, several innovations were incorporated 
to the traditional BMI approach over the past several 
years. In 2009, the Nicolelis laboratory published the 
first BMI approach to decode kinematics of bipedal walk- 
ing in rhesus monkeys (261) (FIGURE 12). Two years 
later, in 2011, the same laboratory implemented, for the 
first time, a method for multi-channel ICMS as a tool to 
deliver direct tactile feedback to the subject’s somatosen- 
sory cortex in a BMI setup (598)(FIGURE 13). This new 
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Neural Data 


Position Trace 


___ -20 
EO 9 Time (sec) 3° 


Monkey 2 


Cc Variable Speed Tracking and Predictions 


Slow Forward Walking, SNR = 6.44 — Actual —— Predicted 


Ankle X Coordinate 


50 


Time (s) 


FIGURE 12. Decoding kinematics of bipedal walking from corti- 
cal ensemble activity. A: diagram of the experimental setup, con- 
sisting of a treadmill, video tracking system, neural recording 
system (Plexon, USA) and a computer for real-time decoding of 
neural activity. B: video frames depicting step cycles of two mon- 
keys. C: tracking (blue line) and decoding (red line) of ankle posi- 
tion at different treadmill speeds. [Adapted from Fitzsimmons et 
al. (261).] 


paradigm was named a brain-machine-brain interface 
(BMBI). In those experiments, rhesus monkeys per- 
formed an active tactile discrimination task using a BMBI 
that both generated motor commands and delivered ar- 
tificial tactile feedback directly to the animal’s brain. To 
perform the exploration task, monkeys had to control the 
movements of a virtual hand, using their motor cortical 
activity alone, to scan the surfaces of up to three visually 


identical virtual objects shown on a computer screen. The 
position of the virtual objects changed with every new 
trial. As the virtual hand touched each virtual object, 
temporally patterned ICMS was applied to the animal’s 
primary somatosensory cortex to mimic the virtual tex- 
tures. To receive a juice reward, the monkey had to find 
an object associated with a particular virtual texture, and 
hold the virtual hand over that object. After a few weeks 
of training, two monkeys learned to perform this task at 
levels similar to those attained when the control signal to 
the virtual arm came from the joystick that the animals 
moved with their own biological hands. 


In parallel with these developments, considerable efforts 
have been made by many laboratories to design better 
real-time decoding algorithms. Krishna Shenoy and his 
colleagues (692) reported a “high-performance BMI” 
that relied on the strategy of flashing potential targets in 
a rapid succession on a computer screen. They recorded 
from small neuronal populations in dorsal premotor cor- 
tex and found that the firing of these neurons reflected the 
target location. Target locations could be decoded from 
these neuronal firing modulations using recording inter- 
vals as short as 250 ms, which allowed the BMI to reach 
an information transfer rate of 6.5 bits/s. The authors 
argued that the observed neuronal responses represented 
the monkeys’ motor preparatory activity for arm reach- 
ing movements rather than merely visual responses to the 
targets, and therefore could be useful for controlling a 
motor BMI that automatically directs the cursor to the 
target once its location is determined. The same group 
developed an improved algorithm for continuous cursor 
control (307). The improvement was achieved using the 
“recalibrated feedback intention-trained Kalman filter” 
that was trained using both the cursor position in screen 
coordinates and an estimate of intended velocity based 
on the relative location of the cursor and target. 


Moving in the same direction, our laboratory achieved con- 
siderable improvement of real-time decoding by employing 
an unscented Kalman filter that used position, velocity, and 
speed as state variables and incorporated nonlinear rela- 
tionships between the neuronal rates and these variables 
(491). More recently, Jose Carmena and his colleagues 
(728, 729) reported an improved temporal resolution for an 
adaptive decoder that modeled spikes as a point process. 
Overall, these efforts resulted in a large variety of BMI 
decoding algorithms from which designers can choose de- 
pending on the requirements for their experimental or fu- 
ture clinical implementations. 


While the research on intracranial BMIs has been con- 
ducted mostly in animals for many decades and only 
recently has started to expand into clinical trials in hu- 
man patients, noninvasive BMIs, pioneered by Jacques 
Vidal in the early 1970s (823, 824) and introduced to 
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clinical practice by Niels Birbaumer (71), have under- 
gone a considerable expansion (14, 74, 246, 358, 815, 
863). BMIs for humans are often referred to as BCIs, 
including both invasive and noninvasive systems. It is 
worth noting that the first publication on a human con- 
trolling a robot with EEG activity dates back to 1988 
(85). In this report, subjects issued start and stop com- 
mands to a robot by closing and opening their eyes, the 
well-known procedure to activate and deactivate alpha 
waves (60, 587). Recently, a motor imagery-based BCI 
was used by the Walk Again Project, an international 
nonprofit research consortium, to allow complete para- 
plegic patients to use EEG to control the start and stop 
sessions of bipedal walking of a lower limb robotic exo- 
skeleton (737). Overall, BCI research yielded many prac- 
tical applications, such as BCIs that use EEG signals to 
control computer cursors (240, 864, 865), computer-as- 
sisted spellers (102, 245, 247, 421, 545, 832), wheel- 
chairs (121, 411, 511, 553, 811, 833), and exoskeletons 
that restore bipedal walking (156, 836). 
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FIGURE 13. Brain-machine-brain interface. A: dia- 
gram of experimental setup. Monkey was seated in 
front of a computer screen showing an avatar arm an 
multiple targets. Motor commands were decoded 
from motor cortex activity. Artificial tactile feedback 
was produced by intracortical microstimulation ap- 
plied to primary somatosensory cortex. B: cortical 
location of microelectrode implants. C: microelec- 
trodes used for microstimulation (accented in red). 
D: avatar arm position for a representative trial. The 
monkey first placed the avatar hand over the unre- 
warded artificial texture (UAT), then ultimately se- 
lected rewarded artificial texture (RAT). Vertical gray 
bars correspond to the periods of microstimulation; 
insets indicate stimulation frequency. E: raster dis- 
play of motor cortex discharges for the same trial; 
spikes were not detected during microstimulation de- 
livery because of the stimulation-induced artifacts. 
Only the periods void of microstimulation were used 
for neural decoding. [Adapted from O'Doherty et al. 
(598).] 


ill. BMI CLASSIFICATION 


A. Classification by Function 


Several BMI classification schemes have been proposed 
heretofore. One scheme classifies BMIs according to the 
physiological function they are intended to emulate. Here, 
BMI systems are commonly categorized as follows: 1) mo- 
tor, 2) sensory, 3) sensorimotor (or bidirectional), and 4) 
cognitive. The recent introduction by our laboratory of 
BMIs that incorporate multiple brains of different subjects 
(614, 616, 657) adds one more BMI class, which we named 
Brainet (657). 


Motor BMIs reproduce motor functions, such as upper 
(114, 152, 817, 852) and lower limb (261) movements or 
whole body navigation (656, 873). Sensory BMIs aim at 
reproducing sensations, while sensorimotor BMIs combine 
the motor and sensory components in a single application 
(57, 597, 598). Cognitive BMIs (19) enable higher-order 
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brain functions, such as memory (62), attention (279, 512), 
and decision-making (343, 564). Finally, Brainets involve 
the implementation of shared BMIs requiring the participa- 
tion of multiple subjects since these systems require the 
combination of electrical activity of multiple brains simul- 
taneously to operate properly (657). Although seemingly 
straightforward, this classification scheme follows the tra- 
ditional labeling of brain areas as motor, sensory, or higher 
order (associative), a simplified description, which is funda- 
mentally wrong in most cases. There is mounting evidence 
against such a parcellation scheme of brain functions and in 
favor of a more distributed mode of information processing 
in the primate cortex (96, 253, 497, 583, 675). Thus sen- 
sory and motor signals are typically multiplexed by cortical 
neurons, including the areas considered to be purely motor 
or purely sensory (470). Indeed, even the primary motor 
and somatosensory cortical areas have been shown to con- 
currently process both motor and somatosensory informa- 
tion (229, 277, 497, 500, 647), and also represent visual 
signals (738), reward amount (529), and even cognitive 
processes, such as mental rotation (297, 300) and encoding 
serial order of stimuli (115). 


Based on our own BMI work, we have argued for more than 
a decade that highly distributed neuronal ensembles repre- 
sent the true physiological unit of the nervous system (583), 
a proposition that is supported by an extensive literature 
(107, 109, 253, 302, 339, 477, 648, 686, 687). Ina distrib- 
uted neural circuitry, there are no exclusive functional spe- 
cializations at the level of individual neurons. Rather, single 
neurons multiplex several functions, and the best decoding 
can be achieved by sampling large numbers of neurons from 
multiple brain areas, simultaneously. We have proposed, 
therefore, that as the BMI field advances, this new theoret- 
ical view of brain function will lead to almost universal 
acceptance of a distributed principle of neuronal activity 
sampling in BMI applications. Since our first BMI studies 
(852), we have consistently demonstrated that large-scale 
neuronal recordings from multiple cortical areas are imper- 
ative for building BMIs that are versatile, robust, efficient, 
and clinically relevant (466, 575, 583). Another step in this 
direction was made when the BMBI paradigm was intro- 
duced (598). In a BMBI, motor and sensory streams of 
information are handled simultaneously to facilitate senso- 
rimotor processing. In the same context, the concept of 
Brainets takes multitasking BMI designs to the next level by 
combining multiple brains into a higher order computa- 
tional entity (614, 616, 657). 


B. Classification by the Level of Invasiveness 


Since the inception of modern-era BMIs, two major ap- 
proaches have dominated the field: intracranial or invasive 
(124, 852) and noninvasive (71) systems. Accordingly, it is 
common to classify BMIs by their level of invasiveness. This 
division is important for a variety of reasons, the major one 
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being a safety concern. Invasive BMIs require a neurosur- 
gery procedure that involves opening the scalp and skull 
and penetrating the brain tissue, albeit only for a few milli- 
meters, for systems relying on cortical signals. These proce- 
dures carry a risk of tissue damage and/or infection, partic- 
ularly if the implant is not fully contained within the body 
and has external parts, like wires connected to extracranial 
recording hardware, as was the case in recent clinical trials 
in humans (83, 152, 360, 362). Noninvasive BMIs, on the 
other hand, do not carry such risks and can be implemented 
rather easily. For example, in the case of EEG recordings, 
the electrodes are simply placed on the scalp surface (588) 
through an easy and safe procedure, particularly if dry sen- 
sors (136, 266, 287, 329, 785) are used. 


Since patient safety is of paramount importance, noninva- 
sive BMIs are currently the default choice for clinical appli- 
cations. Yet, their recording quality is insufficient in many 
cases, causing EEG-based BMIs to be rather slow systems. 
Indeed, EEGs represent attenuated and filtered brain activ- 
ity, which combines synchronous electrical signals, pro- 
duced by many millions of neurons. Since these signals have 
to travel through bone and skin prior to reaching the scalp 
sensors, EEG signals lack fine spatial resolution and do not 
provide the kind of precise task-related neuronal signals 
that can be obtained from intracranial recordings (466). 
Similar limitations characterize all noninvasive recording 
methods that measure neuronal signals at a distance from 
their source. 


In contrast, in invasive BMIs, recording sensors are brought 
close to the very source of generation of neural activity: the 
single neurons that code information through trains of ac- 
tion potentials (157, 318, 487). Current intracranial BMIs 
usually employ extracellular recording methods that allow 
one to sample and discriminate action potentials generated 
by hundreds of individual cortical neurons (466, 583, 711, 
767). The more microelectrodes are implanted; the more 
neurons can be sampled simultaneously. Moreover, the 
same microelectrodes can also record local field potentials 
(LFPs), which represent combined potentials of large (on 
the order of tens of thousands) neuronal populations (192, 
289, 373, 398, 571). Additionally, the same implanted mi- 
croelectrodes that are used for recordings can also be used 
for the delivery of electrical microstimulation that, depend- 
ing on the stimulated area, can influence sensory, motor or 
cognitive processing (41, 45, 48, 146, 704, 705, 719, 770). 


An intermediary approach, known as electrocorticography 
(ECoG) can be considered as a semi-invasive method since it 
requires a craniotomy but does not involve sensors pene- 
trating the nervous tissue. ECoGs are recorded with a grid 
of electrodes placed on the brain’s surface; dura mater may 
be left intact (i.e., epidural ECoG) or open to allow closer 
contact between the electrodes and the cortex (subdural 
ECoG) (24, 94, 357, 530, 665, 696, 828). ECoG recordings 
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have better spatial and temporal resolution than EEG, but 
they cannot be used to reliably detect single-neuron spikes. 
Chronic ECoG recordings can last many years in animals 
(105, 123, 131, 508, 682) and humans (845, 870). Overall, 
ECoG has many advantages compared with EEG, and it is 
not as invasive as penetrating implants. Still, it remains 
controversial whether or not ECoG-based BMIs can rival 
single units in terms of BMI performance and accuracy. A 
variety of intracranial and invasive approaches are re- 
viewed below in the section on neural recording methods. 


C. Classification by the Origin of Neural 
Signal 


Since the late 1990s when the modern BMI concept and 
design was introduced, the majority of BMIs have utilized 
neural signals recorded from cortical areas of animals or 
human subjects (71, 114, 124, 152, 362, 377, 794, 852, 
864). Such an abundance of cortical BMIs is not surprising 
because the cortex is the largest and most advanced brain 
structure, which is also the easiest to access with the record- 
ing sensors. Among cortical areas utilized in BMIs, M1 is 
the most commonly implanted area (114, 725, 852) because 
neuronal discharges in M1 are clearly correlated with dif- 
ferent movement parameters (227, 294, 554). Additionally, 
recordings from premotor cortex are employed in BMIs as a 
source of motor commands to control ongoing movements 
(114) and preparatory signals that reflect motor planning 
before movements have started (692). 


Our laboratory has long advocated recording simultane- 
ously from multiple cortical areas as an efficient way to 
increase both the amount of information processed by a 
BMI and its performance and versatility (114, 377, 466, 
583). We have routinely recorded from four to eight frontal 
and parietal cortical areas in rhesus monkeys to operate a 
variety of BMIs and observed that any of those cortical 
areas provide useful information. The best BMI perfor- 
mance was usually achieved when neuronal signals from 
multiple frontal and parietal cortical areas were combined. 


Lately, interest has increased to BMI systems that utilize 
subcortical recordings. These systems, in principle, could 
capture neural processing in cortico-subcortical loops (11) 
related to motor control (188, 189, 677), sensory process- 
ing (44, 396, 397, 447, 528), motivation (706-709), and 
skill learning (72, 212, 356, 689). It is noteworthy that 
subcortical recordings from the ventrolateral thalamus 
(VL) were utilized in the pioneering BMI study by Chapin et 
al. in 1999 (124) where VL neurons contributed to the BMI 
control of lever movements. Additionally, Patil et al. (620) 
relied on subcortical recordings from the subthalamic nu- 
cleus (STN) and ventral intermediate/ventral oralis poste- 
rior motor thalamus (Vim/Vop) to demonstrate, for the first 
time, that human patients could utilize real-time algorithms 
previously tested in monkeys to generate one-dimensional 
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movements of a computer cursor (336). More recently, Ko- 
ralek et al. (442) employed cortical and striatal recordings 
in awake, behaving rats to construct a BMI for abstract skill 
learning. In these experiments, rats learned to control the 
pitch of an auditory stimulus through a BMI. This learning 
was accompanied by increases in neuronal firing in the stria- 
tum. Additionally, stronger correlations developed between 
the cortical and striatal neurons. These findings further sup- 
ported the claim, made in early BMI studies (114, 124, 
463), that brain plasticity, in this case, corticostriatal plas- 
ticity, plays a major role in learning to control a BMI. This 
is an important conclusion for the development of BMIs, 
since it means that severely disabled patients may be able to 
learn novel motor skills, through BMI training. 


In the future, subcortical BMIs could contribute to treat- 
ment of neural conditions caused by disorders of subcorti- 
cal processing, such as Parkinson’s disease. As a step in this 
direction, we analyzed pathological signs in neuronal pop- 
ulations in Vim/Vop and STN recorded in Parkinsonian 
patients (336), while they controlled a computer cursor by 
opening and closing their hands. Their task was to point to 
screen targets with the cursor. Vim/Vop and STN neuronal 
populations responded to target onset, and hand move- 
ments, as well as being correlated with hand tremor. BMI 
decoders extracted movement kinematics from the STN 
population activity even when those populations exhibited 
tremor-related oscillations. These findings indicate that, in 
the future, BMIs based on subcortical recordings could be 
used for monitoring signs of neurological diseases, evaluat- 
ing medical treatments and even delivering real-time reha- 
bilitation therapies, via implanted devices, without the need 
for continuous supervision. 


D. Classification by BMI Design 


Development during the last two decades resulted in several 
well-established BMI designs. Two broad classes of BMIs 
are represented by the so-called independent (endogenous) 
and dependent (exogenous) systems. Although this termi- 
nology is usually applied to noninvasive BMIs (i.e., BCIs), it 
is also applicable to intracranial BMIs. In an independent 
BMI, subjects self-initiate actions, for example, by imagin- 
ing movements (604, 631, 651, 698) or even assisting them- 
selves with overt movements of the limbs (114, 307, 463, 
852). Such imagery and self-generated movements are con- 
trolled voluntarily by the subjects and, in principle, could be 
performed independently from any external stimuli (al- 
though some external stimuli are usually involved). 


Dependent BMIs, as the name suggests, critically depend on 
the presence of an external stimulus and the triggered neural 
responses to this stimulus (246, 473, 717, 832). For exam- 
ple, a P300 EEG- or ECoG-based BCI monitors cortical 
responses to computer screen events and detects a stronger 
response to the stimulus attended by the subject (101, 209, 
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257, 545, 635, 805). A very similar design, mentioned 
above, was implemented in an intracranial BMI that evalu- 
ated premotor cortex responses to visual targets that rap- 
idly flashed on the screen (692). The improvement in per- 
formance was achieved because the decoder received the 
information at the time of stimuli presentation and started 
to analyze neuronal data precisely after this event. Such 
utilization of the timing of external events is characteristic 
of all dependent BMIs. For example, BMIs that incorporate 
instructed delay tasks rely on a precise sequence of computer- 
generated task events for decoding (564, 731). While such 
supervised operation speeds up the decoding and makes it 
more reliable, dependent BMIs can operate only for a given 
set of rules, a property that clearly limits the user’s auton- 
omy in choosing motor outputs. 


In addition to independent and dependent BMIs, the term 
passive BMI (or passive BCI) has been recently introduced 
to describe a system that performs useful decoding of neural 
signals without considerable mental efforts of the subjects 
(885, 887). Passive BMIs could, for example, improve hu- 
man interactions with a technical system by monitoring and 
decoding neural signals representing cognitive and emo- 
tional states, while making appropriate adjustments to the 
technical system. 


Another criterion to classify BMIs is whether a subject per- 
forms overt movements while performing a motor task us- 
ing the BMI. Early BMI demonstrations relied on overt 
movements to train the decoder and to operate a BMI (114, 
261, 463, 852), whereas the next generation of BMIs ex- 
cluded overt movements from both training and operation 
phases (377, 794). The goal of this latter modification was 
to mimic more properly the conditions of paralyzed patients 
who cannot produce overt movements. Withholding overt 
movements during BMI control dramatically changes neu- 
ronal ensemble activity patterns (463, 583), including a 
transient increase in correlation between cortical neurons, 
which tends to subside with further training (377). 


Overall, having a monkey control a BMI without moving its 
own limbs is a much more difficult task for the animal 
compared with the BMI control assisted by overt behav- 
iors. Some recent studies still allow monkeys to assist 
themselves in the BMI operation by producing overt limb 
movements (239, 307, 761). While these reports down- 
play the significance of this component (307), it is possi- 
ble that the presence of overt movements contributed to 
the improvement in BMI decoding. We suggest that overt 
behaviors should be better monitored in BMI studies and 
compared with the changes in BMI performance. 


In this context, we should also mention that the require- 
ment to withhold limb movements, while producing actua- 
tor movements only through a BMI, defines a particular 
motor task by itself; one that resembles the well-known 


782 


instructed delay task, where cortical neuronal firing modu- 
lations (143, 171, 535, 846, 859), and even spinal cord 
interneurons (653), change their activity in the absence of 
overt limb movements. Similar no-go requirements can be 
found in BMI task designs. For example, Ganguly et al. 
(284) attached monkeys’ arms to a KINARM apparatus 
and required the animals to maintain constant arm position 
on each trial, while moving a screen cursor using a BMI; any 
arm movements cancelled the trial. These experimental set- 
tings are virtually identical to a classical instructed delay 
task, with the exception that in the instructed delay task, the 
arm eventually moves to the target, whereas in the BMI 
task, arm movements were not required. Such a require- 
ment to pay attention to the arm position is different from 
other studies where animals were not encouraged to pay 
attention to the arm position, which was irrelevant for the 
BMI performance (114, 377, 463). This difference is impor- 
tant because the results of such experiments are often inter- 
preted in terms of incorporating an external effector into 
the brain’s own representation of the subject’s body (466). 
To validate such an interpretation, it matters whether the 
subject refocused attention to a new effector or continued to 
attend to the arm position and possibly used it as a reference 
for a BMI-controlled cursor. 


In the early days of the field, the number of neurons 
needed for accurate BMI performance control used to be 
a controversial issue. During the early 2000s, research 
groups led by Donoghue (723, 725) and Schwartz (794) 
suggested that recording from just a few cortical neurons 
simultaneously could be sufficient for achieving good 
BMI control. In contrast, our laboratory has always ar- 
gued that large neuronal ensembles are required to 
achieve optimal BMI performance (114, 261, 377, 466, 
468, 583, 852). We reasoned that single neurons repre- 
sent behavioral parameters of interest only partially and 
in a noisy way, and that combining contributions from 
many neurons both increases the information content 
and improves the signal to noise ratio of decoding. Ad- 
ditionally, it is easier to select neurons with the properties 
needed for decoding when there is a sufficiently large 
neuronal sample from which to select. As time passed, 
this dispute was resolved in favor of the large neuronal 
samples. Even the former proponents of small ensembles 
have now switched their approaches to embrace large 
neuronal samples as the only viable approach for clini- 
cally-relevant BMI applications (152, 360, 362). 


IV. REPRESENTATION OF INFORMATION 
BY NEURONS AND THEIR 
ENSEMBLES 


A. Tuning of Single Neurons 


For the past six decades, an extraordinarily large neuro- 
physiological literature has accumulated around the subject 
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of how ethologically meaningful information is encoded by 
individual neurons and neuronal ensembles (106, 142, 237, 
293, 401, 578, 583, 671). Despite these extensive observa- 
tions, we are still far from understanding the physiological 
mechanisms guiding the dynamical operation of neural cir- 
cuits in mammals. This fundamental lack of knowledge, 
however, has not precluded BMI research from adopting an 
empirical approach, where parameters of interest are ex- 
tracted from neuronal signals. Any decoding approach is 
always based on the existence of some degree of correlation 
between neural activity and those parameters of interest, 
but rarely on an unequivocal establishment of a causal re- 
lationship or a clear knowledge of the neural mechanisms 
involved. Yet, as we previously argued (466, 583), empiri- 
cal approaches employed in BMI research could help us to 
uncover fundamental neurophysiological principles gov- 
erning the operation of brain circuits. 


The existence of a correlation between neuronal firing rate 
and a given behavioral variable is classically referred to as 
neuronal tuning. When one says that a neuron is tuned to a 
behaviorally relevant parameter, this simply means that the 
neuronal rate is correlated with that parameter, and this 
correlation is consistent. In this context, two key physiolog- 
ical properties of neuronal circuits have accounted for the 
feasibility of creating functional BMIs which can generate 
consistent motor outputs. First, BMIs have benefited from 
one of the early discoveries of neural ensemble physiology: 
that the trial-to-trial variability in firing rates of single neu- 
rons, which is often described as neuronal noise, can be 
significantly compensated by recording simultaneously 
from ensembles of neurons (253, 292, 462, 467, 583). In 
other words, combining contributions from many neurons 
reduces the uncorrelated noise produced by individual neu- 
rons while leaving intact the consistent component of firing 
modulations, best represented by the entire neuronal pop- 
ulation. The second physiological property that proved to 
be indispensable for the proper operation of BMIs is the 
occurrence of neuronal plasticity, i.e., the ability of neurons 
to continuously adapt their tuning when exposed to novel 
task contingencies and external world statistics (see below). 


The relevant literature on neuronal tuning starts with the 
work of Edward Evarts who pioneered the investigation of 
the physiological properties of single M1 neurons in the 
mid-1960s (233, 235, 236). By using sharp-tip electrodes, 
Evarts sequentially recorded extracellular electrical activity 
of single M1 neurons, while his monkeys remained awake 
and performed a variety of motor tasks (227, 228, 230- 
232, 234, 238). These now classical experiments revealed 
that M1 neurons were tuned to parameters such as muscle 
force and joint torque. For example, an M1 neuron would 
increase its firing rate when the monkey pulled a lever and 
decrease firing when the monkey pushed it. As such, M1 
neuronal firing modulations coded the next push or pull 
movements performed in a task trial. Combining neuronal 
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firing rates from many trials of each kind allowed Evarts to 
describe an average response curve, which was named 
perievent time histogram (PETH). 


Evarts’ original findings triggered a major push in systems 
neurophysiology, resulting in the widespread use of his 
technique for single-unit recording to characterize the 
tuning properties of individual neurons in various areas 
of the rhesus monkey’s brain. In the case of the motor 
cortex, the next breakthrough was produced by the sem- 
inal discovery of Apostolos Georgopoulos and his col- 
leagues that M1 neurons exhibit broad tuning to the 
direction of arm movement (295, 299, 423, 710). To 
reach this conclusion, the Georgopoulos laboratory mea- 
sured the firing discharge patterns of individual M1 neu- 
rons, while their monkeys performed arm reaching move- 
ments that started at an initial, central location and 
ended on peripheral targets (FIGURE 14). Analysis of 
these center-out movements revealed that the single neu- 
ron’s firing rate peaked for a particular movement direc- 
tion, called preferred direction, and decreased gradually 
when the direction deflected from the preferred one. 
Georgopoulos graphically represented the relationship 
between neuronal rate and movement angle as the direc- 
tional tuning curve (290, 710). He also proposed a cosine 
fit, where the neuronal rate was proportional to the co- 
sine of the angular deviation from the preferred direc- 
tion. 


Further exploration conducted by many groups into the 
activities of cortical neurons during motor behaviors re- 
vealed many additional characteristics of neuronal tuning, 
such as representation of sensory signals (464, 505, 565), 
sensorimotor transformations (194, 297, 298, 400, 401, 
860), simultaneous encoding of the motor goal and the 
direction of spatial attention (471), representation of mul- 
tiple motor plans (143), and even encoding of cognitive 
variables by M1 neurons (291). 


With the accumulation of these findings, it became progres- 
sively clear that the next step toward the understanding of 
brain encoding would require sampling the activity of pop- 
ulations of neurons recorded simultaneously, instead of re- 
cording one neuron at a time. In the case of BMIs, this step 
was imperative because single neurons or small neuronal 
samples (10-30 neurons) could not sustain BMI opera- 
tional accuracy and stability. 


B. Neuronal Ensemble Physiology 


At present, it has become conventional for BMI studies to 
report that decoding of behavioral parameters from neuro- 
nal activity improves with an increase in neuronal sample 
(114, 261, 377, 463, 466, 468, 583, 852). The relationship 
between the sample size and decoding accuracy is depicted 
by a NDC (FIGURE 11), an analysis introduced by our lab- 
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FIGURE 14. Broad tuning to movement direction of a motor cortex neuron. In this study of Apostolos 
Georgopoulos and colleagues, monkeys performed center-out arm movements in different directions. A: raster 
displays of the neuron activity for eight movement directions. B: arm trajectories. C: directional tuning curve 
showing neuronal firing rate as a function of movement angle. [Adapted from Georgopoulos et al. (294).] 


oratory in 2000 (852). Following this original publication, Based on a decade of BMI studies in our laboratory, we 
this analysis quickly became a standard in the literature on have proposed a series of principles of neural ensemble 
neural decoding (50, 78, 283, 552, 564). physiology, derived primarily from the analyses of NDCs 
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Table |. 


Principle 


Distributed coding 

Single neuron insufficiency 
Multi-tasking 

Mass effect principle 


Principles of neural ensemble physiology 


Explanation 


Representation of any behavioral parameter is distributed across many brain areas 
Single neurons are limited in encoding a given parameter 

A single neuron is informative of several behavioral parameters 

A certain number of neurons in a population is needed for their information 


Capacity to stabilize at a sufficiently high value 


Degeneracy principle 
Plasticity 


The same behavior can be produced by different neuronal assemblies 
Neural ensemble function is critically dependent on the ability to plastically adapt to 


new behavioral tasks 


Conservation of firing 


Context principle 
stimulus 


Overall firing rates of an ensemble stays constant 
Sensory responses of neural ensemble changes according to the context of the 


[From Nicolelis and Lebedev (583).] 


(583, 586) (TABLE 1). Several of these principles can be 
immediately derived from the typical NDC, which shows a 
rapid rise in decoding accuracy for small neuronal samples 
followed by a slower rise for larger populations. Accord- 
ingly, the single-neuron insufficiency principle states that 
individual neurons carry low amounts of information. Con- 
versely, the neuronal mass principle states that a given neu- 
ronal ensemble should reach a certain size, in terms of num- 
ber of neuronal elements, for decoding accuracy to stabilize. 
At this point, decoding performance does not change sub- 
stantially when a few neurons are added or removed. This 
stabilization of decoding accuracy should not be confused 
with saturation. Further improvement of decoding is possi- 
ble, but a substantial increase in neuronal sample is needed 
to achieve an appreciable effect. The level of information 
grows as a function of the logarithm of the neuronal sample 
size. FIGURE 15 shows NDCs for several cortical areas and 
a variety of BMI experiments conducted in rhesus monkeys. 
It follows from the analysis of these NDCs that typically 
each cortical area contains some level of useful information, 
regarding the decoding of a given motor parameter. Ac- 
cordingly, the distributed-coding principle postulates that 
information is represented by the cortex in a distributed 
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way. This means that, for a given behavioral parameter, 
neurons distributed within multiple cortical areas partici- 
pate in the representation and processing of that parameter. 
As an illustration, a study that employed multielectrode 
recordings from dorsal premotor cortex (PMd) and ventral 
premotor cortex (PMyv) (68) showed that both areas repre- 
sented the kinematics of arm reaching and characteristics of 
hand grasping, the conclusion that favored the distributed- 
coding principle and rejected a dual-channel hypothesis 
that attributed the representation of reaching to PMd and 
the representation of grasping to PMv (673). 


Next, the neuronal multitasking principle proposes that an 
individual neuron can simultaneously represent multiple 
behavioral parameters, for example, arm kinematics and 
gripping force exerted by the hand (114). Consistent with 
this statement, a recent review by Fusi et al. (280) confirmed 
that neuronal responses often represent combinations of 
behavioral parameters. Fusi et al. (280) suggested that such 
high-dimensional representation is computationally advan- 
tageous compared with a population of highly specialized 
neurons because it allows the nervous system to generate a 
huge number of potential responses from the inputs to the 


Gripping Force 


20 
Number of Neurons 


FIGURE 15. Neuronal dropping curves 
(NDCs) for the decoding of arm velocity 
and hand gripping force in rhesus mon- 
keys. NDCs were calculated separately 
for the neuronal populations recorded in 
different cortical areas: dorsal premotor 
(PMd), primary motor (M1), primary so- 
matosensory (S1)], supplementary motor 
(SMA), and posterior parietal (PP). A: de- 
coding of velocity. B: decoding of gripping 
force. [From Carmena et al. (114).] 
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neuronal population. Indeed, our neural degeneracy princi- 
ple supports the hypothesis of Fusi et al. by proposing that 
a given neuronal ensemble can generate many behavioral 
outputs and, conversely, that a given behavioral output can 
be encoded by a variety of neural ensembles. According to 
our view, such a principle, which is similar to degeneracy 
observed in the genetic code (221, 853), improves the ro- 
bustness and flexibility of neural encoding (480). 


An additional clarification of the relationship between neu- 
ronal ensemble physiological patterns and associated be- 
haviors is provided by the conservation of firing principle. 
This principle explains that, due to the metabolic constraint 
created by a fixed brain energy budget, the overall discharge 
rate of a neuronal population should always remain con- 
stant. Thus, if some cells increase their firing rate to encode 
a motor parameter, others should reduce their activity pro- 
portionally, to allow the energy consumption to remain 
around a set limit. This principle is closely related to the free 
energy principle which states that the brain minimizes its 
free energy when exchanging information with the environ- 
ment (273-275, 806). 


Within the constraints of the conservation of firing princi- 
ple, the context principle states that information represen- 
tations by neural ensembles depend on behavioral context. 
Neuronal response to an event would be different depend- 
ing on the set of circumstances that surround the event, for 
instance, whether the animal is awake or anesthetized when 
the same sensory stimulus is delivered. Differences in neu- 
ronal firing patterns produced under these two different 
conditions, according to our view, reflect the brain’s ability 
to contextualize information (527). Finally, the plasticity 
principle states that neuronal populations modify their 
properties when an organism adapts to novel conditions or 
learns new behavioral tasks, including BMI tasks. 


Our findings from BMI studies are consistent with these 
neural ensemble principles. From more than a decade of 
BMI research, we have learned that the best and most op- 
timal way to extract a motor parameter from brain signals 
is to rely on concurrent recordings of the activity of large 
populations of neurons, distributed across multiple cortical 
areas. Clearly, this sample should include a large popula- 
tion of neurons located in M1, the cortical area from which 
most reliable kinematic and dynamic information can be 
obtained. Yet, surprisingly, many nonprimary frontal and 
parietal areas can contribute meaningfully to a BMI that 
continuously controls movements of an external actuator 
(3, 114, 852). On the basis of these findings, it is fair to say 
that BMI research has contributed decisively to consolidate 
a new view of cortical processing, one that departs from the 
classical dogma, proposed by the neuron doctrine, where a 
single neuron is considered the true functional unit of the 
brain, to one in which distributed neuronal populations 
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assume that key physiological role. Put in other words, BMI 
research truly vindicated the Hebbian view of the brain. 


Among multiple contributions to a better understanding of 
cortical ensemble physiology (583), BMI research has led to 
several demonstrations of new types of cortical plasticity. 
Indeed, since the early BMI studies, it became clear that it 
was only through cortical plastic adaptations that subjects 
could learn to control a BMI and improve their overall 
motor performance over time (114, 283, 377, 794). Even- 
tually, this cortical plasticity led to the assimilation or in- 
corporation of external actuators, such as robotic arms and 
legs, as if they were true extensions of the subject’s body 
representation that is known to exist in the brain (466, 
583). BMI-associated cortical plasticity is manifested by 
changes in directional tuning patterns of individual neurons 
(114, 283, 463), alterations in temporal patterns of neuro- 
nal discharges (884), and a transient increase in the corre- 
lation between neurons within and between multiple fron- 
tal and parietal cortical areas (114, 377, 583). All these 
physiological adaptations mean that neuronal space in the 
cortex becomes devoted to representing a variety of prop- 
erties associated with the artificial actuators employed by a 
BMI. 


V. MULTICHANNEL RECORDING 
TECHNOLOGY 


A. Microwire Recording Cubes 


Developing methodology for reliable recordings from large 
populations of brain neurons is essential for further ad- 
vances in our understanding of neuronal ensemble physiol- 
ogy and for the development of more practical BMI appli- 
cations (466, 581, 583, 711). So far, the major achieve- 
ments in this field have been associated with multielectrode 
implant methods. 


We start by discussing the technology employed by our 
laboratory for the past 25 years because it has produced the 
highest neuronal yield and postimplant longevity reported 
in the literature so far (711). Thus, in our hands, a typical 
multielectrode implant was originally defined, in the mid- 
1990s, by a two-dimensional grid composed of small-diam- 
eter (12-50 wm) Teflon or isonel-insulated metal mi- 
crowires (452, 581, 711). This technology evolved over 
nearly three decades of research, through experiments in 
rats, mice, monkeys, and human subjects. Optimal param- 
eters were then obtained empirically over years of experi- 
mentation. Our latest configuration of this technology is 
defined by so-called volumetric, movable implants, named 
recording cubes, introduced by our laboratory over the past 
3 years (FIGURE 2) (711). A recording cube is built by first 
constructing a grid of polyimide guiding tubes. The grid has 
a 10 X 10 arrangement with 1 mm spacing between the 
adjacent tubes. This spacing is optimal for monkey record- 
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ings; denser grids can be harmful to cortical tissue. The 
guiding tubes are contained in a three-dimensional printed 
plastic case that also holds an array of miniature screws 
utilized to move the microelectrodes. Each guiding tube 
accommodates 3-10 microelectrodes. These microelec- 
trodes can have different lengths, which allow their place- 
ment at different depths in both cortical and subcortical 
areas. For that reason, these cubes allow us to obtain volu- 
metric recordings from whatever cortical/subcortical struc- 
ture is sampled. 


During penetration into the brain, the microwires con- 
tained inside each tube move as a single set. This set 
includes the longest, leading microelectrode that pene- 
trates the brain first. The leading microelectrode has a 
conical tip, whereas the other microwires in the set have 
either conical or cut angle tips. This configuration of the 
tip shape allows each microwire set to penetrate through 
the monkey pia. Each recording cube is compact (surface 
area of 0.22 mm? per recording channel) and light (11.6 
g). Since a 10 X 10 tube grid can contain up to 10 mi- 
crowires per tube, each recording cube provides a total of 
1,000 potential recording sensors. We typically record 
1-2 individual neurons per microwire, which provides a 
potential sample of 1,000-2,000 neurons per recording 
cube implanted. Since we usually implant 4-8 such re- 
cording cubes in a single monkey, the potential neuronal 
sample that could be recorded in each monkey ranges 
between 4,000 and 16,000 neurons. 


Although in the past we experimented with a variety of 
microelectrode materials, for example, tungsten for the 
shafts and gold-plated tips (478), we currently use only 
polyimide insulated stainless steel microelectrodes, 30-50 
pm in diameter for recordings in monkeys, because they 
were found to have the best longevity and quality of record- 
ings. For recordings in rats, tungsten microwires are useful 
because they allow recordings for up to 6 mo and do not 
induce any significant neuronal death or tissue inflamma- 
tion, although recording quality deteriorates with time due 
to glial encapsulation of the microelectrodes (271). 


The quality of the implantation surgery is a key factor in 
defining the recording array performance and its long-term 
reliability (607). In our laboratory, primate surgery is con- 
ducted in strict sterile conditions under general anesthesia. 
The animal is placed in a stereotaxic apparatus, and 
craniotomies are made over the areas of interest. Dura ma- 
ter is removed inside the craniotomy, and the guiding tubes 
are placed in light contact with the pia mater. The recording 
cube is then fixed to the skull with dental cement; stainless 
steel and ceramic bone screws serve as anchors. The implant 
is then encased in a three-dimensional printed protective 
cap, which can also house the components of our wireless 
recording system (FIGURE 2). 
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Microelectrode penetrations are performed 1-2 wk after 
the surgery under ketamine anesthesia. Generally, it takes 
1-2 wk to insert 500-1,500 microelectrodes in each animal. 
The penetrations are performed slowly and over multiple 
days to minimize tissue damage. Importantly, during each 
penetration, only a small number of microelectrodes are 
moved to avoid dimpling of the cortex and a bed of nails 
effect, where an electrode array cannot penetrate because 
the pressure is evenly distributed among many electrodes. 
Depending on the recording cube design, rotation of one 
miniature screw brings in motion the microelectrodes lo- 
cated in 1-4 guiding tubes. Electrophysiological recordings 
are conducted simultaneously with each penetration to con- 
firm that the microelectrodes gradually move through the 
cortical layers as the miniature screws are rotated. We usu- 
ally penetrate the cortex with the microelectrodes spaced no 
closer than 2 mm, and perform penetrations with other 
microelectrode subsets on a different day. Once the micro- 
electrodes are placed in their designated locations, they are 
never moved afterwards. Several months later, the guiding 
tubes are encapsulated by connective tissue, and eventually 
the spaces of the craniotomies may get ossified. 


We usually place recording cubes over both hemispheres, 
including areas such as the primary motor (M1), primary 
somatosensory ($1), dorsal premotor (PMd), supplemen- 
tary motor (SMA), and posterior parietal (PP) cortical ar- 
eas. With this approach to cortical recordings, we have 
routinely recorded from 300-1,700 neurons in a single an- 
imal per day; good recording quality and high neuronal 
yield typically have continued for several years (FIGURE 16). 
Indeed, at the time of this writing, two of our monkeys 
completed more than 7 yr of cortical recordings after the 
original implantation surgery. The same microelectrodes 
can be used for both electrophysiological recordings and 
electrical microstimulation for many months (597-599). 
Viable recordings with implanted microwire arrays for 7 yr 
were also reported by the Rizzolatti laboratory (453). 


We recently designed multielectrode implants suitable for 
chronic implantation into subcortical structures in rhesus 
monkeys, such as the neostriatum, thalamus, and the hip- 
pocampus. In this design, subsets of 10-30 microwires 
form bundles, staggered at 1-1.5 mm, which are inserted 
through individual guiding tubes. Each guiding tube is in- 
serted into the brain at a depth of 5-15 mm (depending on 
the target structure). The microelectrode bundle is passed 
through the length of the guiding tube, and then travels an 
additional 5-10 mm to cover the area of interest. 


B. Utah Array 


While multielectrode implants employing individually mov- 
able microwire bundles can be placed at a wide range of 
cortical and subcortical depths, several laboratories have 
used an array constructed of 100 rigid microelectrodes in a 
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FIGURE 16. Number of neurons recorded by microwire implants over time. The graph shows the average 
number of units sampled per 32-channel connector. Data from eight monkeys are presented. Sample 
waveforms are shown for one of the monkeys (monkey M), for different dates after the implantation surgery. 


[From Schwarz et al. (711).] 


fixed arrangement, known as the Utah array or Utah probe 
(111). The array is micromachined from silicon. Each sili- 
con needle is ~1.5 mm long. The needles’ shafts are coated 
with polyimide, whereas their sharpened tips are coated 
with platinum. The spacing between neighboring needles is 
0.4 mm. Insulated gold wires make electrical contacts to the 
back sides of the needles. 


Originally, attempts at slowly lowering the Utah array 
into cortical tissue failed because such a dense array pro- 
duced cortical dimpling, due to the bed of nails effect, 
resulting in an incomplete penetration of some of the 
individual needles (111). To overcome this problem, the 
inventors of this device adopted an impact insertion pro- 
cedure whereby the array is pushed into the cortex at a 
high speed, though a pneumatic gun (680, 681). An ex- 
amination of the long-term recording performance of the 
Utah probe in the cat cortex showed that, 6 mo after the 
implantation, 40% of the needles could not record neu- 
ronal activity, most likely because of fibrous encapsula- 
tion (679). Indeed, extensive fibrous tissue was detected 
on the explanted probes. 


788 


Currently, the Utah array is the only microelectrode im- 
plant approved by the United States Food and Drug Admin- 
istration (FDA) for human use. The array has been em- 
ployed in several clinical studies that involved examination 
of epileptic patients (594, 808, 850) and BMI operation (3, 
83, 152, 360, 361). Yet, instead of sampling single-unit 
activity, most of these studies employed recordings of mul- 
tiunit activity defined as electrical signals that crossed a 
certain voltage threshold. The degree of contamination of 
this signal by electrical and mechanical artifacts was not 
reported, and the possibility of electrical cross-talk between 
the channels was not analyzed. In defense of the poor qual- 
ity of recordings (single-unit recordings are considered a 
gold standard in the field), an argument was put forward 
that simple threshold-crossing is sufficient for BMI control 
(134). While this practical consideration could be valid for 
some implementations, the threshold-crossing method is 
prone to confusing noise and artifacts with real neuronal 
activity. That can easily lead to the undesirable outcome 
that artifacts contaminate the signal used for decoding and 
controlling artificial actuators. These are very real possibil- 
ities that could occur in clinical studies of BMIs, particularly 
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in those in which human subjects, who suffer from quadri- 
plegia, can still move their heads. Artifacts generated by 
head muscle contractions or overall head movement could 
be accepted by a threshold-crossing method as representing 
valid neuronal signals. Supporting this possible scenario, a 
curious example of controlling a BMI by EMG artifacts 
sampled by an EEG cap was recently reported (160). 


Several biocompatibility issues have been reported for Utah 
arrays. The available data on human cortical tissue re- 
sponses to Utah arrays indicate that these implants cause 
tissue reactions, such as microhemorrhaging, microglia ac- 
tivation, and long-term inflammation with the level of se- 
verity depending on the tissue damage during the implan- 
tation (251). Some of these unwanted effects could be 
caused by micro-movements between the brain and the nee- 
dles. In addition to these biocompatibility issues, the Utah 
probe is suitable for recordings only from flat cortical sur- 
faces, not from the sulci. For the flat cortical surfaces, re- 
cordings cannot be obtained from sites located deeper than 
1.5 mm. Because of these shortcomings, the Utah probe 
cannot be considered as the final solution for human im- 
plants despite its current use in clinical trials. There is a 
growing consensus in the literature that research should 
continue into developing better recording technologies suit- 
able for humans (2.51, 271, 391, 624, 646). 


C. Improving Multielectrode Implants 


While microwire arrays currently represent the most prac- 
tical solution for large-scale, multi-area recordings from 
both cortical and subcortical structures, there is ongoing 
research into new technologies. One direction of this re- 
search is aimed at minimizing the micro-movements be- 
tween the implant and the brain that could result in tissue 
damage and inflammation. One solution is to have the im- 
plant float with the brain instead of being anchored to the 
skull. In the design, pioneered by Gualtierotti and Bailey in 
1968 (325), a lightweight implant is tethered by a flexible 
cable. Several implementations of this idea have been devel- 
oped (563, 574, 758). 


Another solution to decrease the motion between brain tis- 
sue and microelectrodes is offered by polymer-based micro- 
electrodes that are more flexible than microwires and exert 
less strain on the brain tissue (4, 344, 451, 788). Such flex- 
ible arrays require specialized insertion techniques that tem- 
porarily stiffen the microelectrode shafts with biodegrad- 
able materials (344, 441) or polymers (754, 789), which are 
then dissolved in the tissue. Promising results were recently 
obtained using a sinusoidal probe anchored to a three-di- 
mensional spheroid tip to minimize the movements between 
the probe and the brain (754). The other type of flexible 
electrodes contains magnetic materials (214, 383). These 
electrodes are inserted using an external magnetic field. 
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Several microelectrode designs have explored the possibility 
of placing recording points along an electrode shaft to in- 
crease neuronal yield and achieve recordings from a volume 
of the nervous tissue. This design is exemplified by the 
NeuroNexus array, also known as the Michigan Probe (23, 
567, 822, 844). This is an array composed of silicon-based 
planar electrodes with multiple recording sites. The number 
of electrodes per shank and the number of shanks can be 
configured. The array can be used for recordings from local 
populations of neurons and for recordings from different 
cortical and hippocampal layers simultaneously (174, 548). 
Recordings with these arrays remain good during the early 
recording days, after which the recording quality usually 
deteriorates (409, 695). 


Tetrodes represent a popular solution to improve the qual- 
ity of single-unit isolation. A tetrode is composed of four 
twisted microwires with blunt tips (389, 664, 857). The 
microwires have differing lengths, which allows sampling 
neuronal potentials in a small three-dimensional space sur- 
rounding the microwire tips. Each neuronal waveform sam- 
pled by a tetrode yields a different amplitude for each mi- 
crowire, due to the differences in distance from the neuron 
to each of the four microelectrode tips. This simple geomet- 
rical arrangement allows better sorting of single units based 
on the extra spatial sampling dimensions added to the pro- 
cess. Recently, tetrode designs have been applied to multi- 
area recordings in primates, where a microdrive advanced 
up to six guiding tubes containing tetrodes to several brain 
areas in awake rhesus monkeys, including primary motor 
cortex, prefrontal cortex, neostriatum, and hippocampus 


(693). 


Ongoing research on new materials could further improve 
both microelectrode recording properties and biocompati- 
bility. Keefer et al. (416) reported that coating of tungsten 
and stainless steel electrodes with carbon nanotubes im- 
proved the recordings by decreasing the microelectrode im- 
pedance and enhancing charge transfer. Suyatin et al. (779) 
developed an electrode based on gallium phosphide nano- 
wires with the sensor composed of a deposited metal film. 
Overall, nanomaterials are a promising research area be- 
cause they can provide better biocompatibility (66, 433, 
446), spatial resolution (215, 799), and electrical properties 
(32, 118). Using pure carbon nanotube probes is another 
promising step in this development (880). 


D. Neurotrophic Electrode 


Neurotrophic electrodes were developed by Philip Kennedy 
in the late 1980s to early 1990s (418, 420, 422) as an 
endeavor to produce a long-lasting solution for the record- 
ings of brain activity in paralyzed and locked-in patients to 
aid the communication of these patients with the external 
world. The idea was to make these electrodes biocompat- 
ible using neurotrophic factors that evoke growth of neu- 
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rites into the recording tip. The electrode was made of a 
hollow glass cone that contained three or four golden, Tef- 
lon-insulated wires glued to the cone walls. The cone was 
inserted in the cortex at a 45° angle to the surface. Histo- 
logical analysis of rat (418) and monkey (420) implants 
showed that neurites grew into the cone and became my- 
elinated. The tissue inside the cone contained axons, axo- 
dendritic synapses, blood vessels, and oligodendrocytes, 
whereas no microglial cells were detected. Bipolar record- 
ings were conducted from pairs of wires. Kennedy and his 
colleagues reported that the cone electrode remained func- 
tional during the entire implantation period, which was 15 
mo in the monkey (420, 422), 16 mo in the rat (418), and 
longer than 4 yr in humans (47). 


Several BMIs were implemented with this implant. A pa- 
tient suffering from brain stem stroke received the implant 
in an unspecified cortical area and was able to control a 
computer cursor with the recorded signals (419). The re- 
cordings continued for more than 17 mo. The patient had 
residual facial movements and eye movements with nystag- 
mus. Correlation of the recorded signals with mouth, face, 
and eye movements were noticed over the 4 mo following 
the implantation. The patient stopped making these move- 
ments afterwards and learned to control the X direction of 
a computer cursor. The cursor moved to the right when 
the recorded signal increased, but did not move when the 
signal decreased. After the cursor reached the right edge 
of the screen, it returned to the leftmost position and 
shifted downward (i.e., a carriage return). With this sim- 
ple control, the patient improved in two tasks: 1) moving 
the cursor to a screen icon and staying on it for 2 s to 
produce synthetic speech using phrases such as “Hello, 
my name is JR,” “I feel uncomfortable,” “I feel too 
cold,” “I feel too warm,” “Please help me,” and “I am in 
pain”; and 2) using a screen keyboard to spell phrases 
that are printed on the screen or vocalized by a speech 
synthesizer. The patient reached a spelling rate of three 
letters per minute. Kennedy and his colleagues claimed 
the patient could dissociate neural activity from the facial 
EMG activity when controlling the cursor. It was also 
implied that the recorded signal was not contaminated by 
the EMGs and other artifacts. 


In the next study, a neurotrophic electrode was placed in a 
speech-related area of the left precentral gyrus of a 
locked-in patient (327). The patient was paralyzed, with the 
motor output limited to the ability to produce slow vertical 
eye movements. The recorded potentials were transmitted 
using a wireless link to a speech synthesizer. The neuronal 
rates were converted into formants that represented small 
sets of continuous sounds (1-s long vowels “uh”, “iy”, “a”, 
or “oo”) using a Kalman filter. The sound served as audi- 
tory feedback. Aided with this feedback, the patient 
achieved a success rate of up to 70% ona three-vowel task. 


Notwithstanding the significance of these results, the exact 
nature of electrical potentials recorded by the neurotrophic 
electrode remains unclear. From the information provided 
in these human studies, it is difficult to tell whether the 
signal contained neuronal spikes or field potentials picked 
by the microwires. To date, no other group reported using 
these or similar recording methods. 


E. Neural Dust 


Neural dust is a recording method that utilizes small (10- 
100 pm) sensors (“dust”) that detect extracellular neuronal 
potentials and communicate them via an ultrasonic link to 
an interrogator placed under the skull (720). Each sensor 
contains a set of electrodes for recording neuronal activity, 
metal-oxide-semiconductor (CMOS) circuitry that ampli- 
fies the signal, and a piezoelectric transducer that converts 
electrical potentials into ultrasound. The interrogator uses 
ultrasound to both power the dust particles and examine 
their state. The interrogator also communicates with the 
extracranial components of the system. 


While the main advantage of neural dust is the absence of 
microelectrode shafts that could be traumatic to the ner- 
vous tissue, the methodology has not been developed yet for 
injecting these particles into the brain. Additionally, funda- 
mental concerns remain regarding effects of implantation, 
signal quality, separation of multichannel signals, and re- 
cordings longevity. There have been no recordings yet of 
cortical potentials using these devices, only recordings of 
large compound nerve potentials and EMGs (721). There- 
fore, the viability of this new method for cortical and sub- 
cortical recordings still needs to be demonstrated. 


F. Endovascular Electrodes 


Using brain blood vessels as entry points for brain recording 
probes is an attractive possibility because this method could 
allow placing the recording sensors close to neurons with- 
out breaking the blood-brain barrier, by inserting thin elec- 
trodes in capillaries (507). 


Endovascular stent electrodes have been used in cardiology 
for several decades. For example, Mirowski et al. (546) 
developed in 1980 an endovascular defibrillator for moni- 
toring cardiac electrical activity and delivering defibrillating 
discharges when ventricular fibrillation was detected. Since 
that time, several modifications of such an endovascular 
cardiac electrode have been proposed (699, 700). 


The same electrode insertion methods have been used for 
endovascular recordings of neural activity (84, 350, 712, 
797). For example, Boniface and Antoun (80) recorded 
EEG activity using a Teflon-coated endovascular guide wire 
that was inserted in the middle cerebral artery in human 
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epileptic patients undergoing preoperative carotid artery 
assessment. 


Recently Oxley et al. (613) conducted multichannel record- 
ings with an array of stent electrodes, called stentrodes, 
which they implanted into the superficial cortical veins 
overlying the sheep motor cortex. Blood vessels as thin as 
1.7 mm in diameter were implanted with the stentrodes, 
yielding brain signals that were comparable to epidural 
ECoG recordings. Recordings in freely moving animals 
continued for 190 days. The authors proposed that such 
their endovascular system could be used to detect seizures in 
epileptic patients, operate BMIs, and deliver electrical stim- 
ulation. 


Nanoscale electronics is a promising method for recordings 
from brain capillaries (507). Masayuki Nakao, Rodolfo 
Llinas, and their colleagues developed nanotechnology 
probes composed of insulated Wollaston platinum wires, 
0.6 wm in diameter (507, 842). The feasibility of these re- 
cordings was demonstrated in the frog spinal cord. The 
sensors were introduced to the bloodstream through a poly- 
imide tube, 90-300 um in diameter. The wires “sailed” 
within a blood vessel until they straightened and could be 
used for recordings. 


G. Optical Recordings 


Optical imaging is based on voltage-sensitive (322, 323, 
621, 793) and calcium-sensitive (320, 748, 771) fluorescent 
dyes, and genetically encoded calcium indicators (516, 523, 
892) whose signals are monitored using video recordings. A 
particularly powerful method, two-photon excitation laser 
scanning microscopy, allows minimally invasive, three-di- 
mensional sampling with submicrometer resolution (353, 
591, 782). Although these techniques require filling neu- 
rons with a dye, hardly a practical procedure for human 
clinical applications, several optical imaging-based BMIs 
have been already demonstrated. 


Clancy et al. (145) conducted experiments in mice with 
genetically encoded calcium indicator gCaMPéf in layers 2 
and 3 in M1 or S1. Recordings were conducted using two 
photon imaging. Head-fixed mice were trained to control 
the pitch of a sound by modulating activity of ~20 optically 
recorded neurons. It took the animals 8 days to learn to 
control sound generation with this BMI. 


Ziv et al. (896) performed calcium imaging in freely behav- 
ing mice whose hippocampal tissue was virally infected and 
coexpressed GCaMP3 and CaMKII in the same neurons. A 
miniature (1.9 g) integrated fluorescence microscope (304) 
was employed for two photon imaging. While the mice 
explored their environment, place fields of thousands of 
hippocampal neurons were tracked over several weeks. 
Next, Bayesian decoding was employed to reconstruct the 
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animal location within the environment from the optical 
recordings. The decoder was trained on the recording data 
collected on one day, and tested on the data the same day or 
different days. The decoding accuracy was the highest for 
the same day, but declined modestly for the different days. 
The authors explained these results by the dynamical 
changes in the hippocampal place fields. 


H. Electrocorticographic Grids 


ECoG grids, containing several tens to several hundred elec- 
trodes, allow minimally invasively recordings of multichan- 
nel field potentials from large cortical territories (172, 357, 
483, 544). 


A recent trend in this approach was to miniaturize the elec- 
trodes and increase their density (79, 542, 825, 838, 895), 
leading to improvements in spatial resolution of the record- 
ings. The efficiency of recordings with high-density ECoG 
can be improved using electronics embedded in the grids. 
For example, Viventi et al. (825) developed ECoG grids 
with embedded flexible nano-membrane transistors that 
performed amplification and multiplexing. This technology 
reduced the number of connecting wires while increasing 
the number of recording channels to several thousands. 


Fu et al. (278) developed flexible mesh electronics with 
micrometer components and bending properties matching 
those of neural tissue. The mesh consisted of 16 electrodes 
and could be injected in the mouse brain using a syringe. 
The implant yielded stable recordings of LFPs and single- 
unit activity for 8 mo. The same probe was used for long- 
term electrical stimulation. 


I. Amplification, Processing, and 
Transmission of Neuronal Activity 


A modern neuronal recording system typically includes sev- 
eral signal processing components. The preamplifier is usu- 
ally placed near the subject’s head and is often called the 
headstage. It performs an initial amplification (typically 
with the gain from 1 to 20) and decreases the output im- 
pedance. This preamplification is needed to reduce the noise 
added at the next signal transmission stage. Depending on 
the system, the headstage output remains analog (601) or is 
digitized (135, 340, 711). The headstage may also perform 
signal multiplexing for the reduction of the number of ca- 
bles in a tethered system. The headstage is connected, via a 
tethered or wireless link, to an external processing unit that 
performs further amplification and/or filtering. Next, the 
neural signals are digitized if they were not digitized by the 
headstage. The digitized signals are then sent to a computer 
for further processing, including neural decoding for run- 
ning BMI tasks. 


While tethered systems were used in early BMIs (114, 124, 
725, 794, 852) and are employed in human clinical trials (3, 
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83, 360), the increasing numbers of simultaneously sampled 
neural channels, the need to study BMIs while animals freely 
behave and the eventual goal of making BMIs fully implant- 
able have prompted the development of wireless technologies. 
FIGURE 2 shows our recently developed wireless BMI, which 
has a 512-1024 channel capacity (711). This system includes 
four modules: 1) digitizing headstages housed in a monkey 
headcap, 2) wireless transceivers also housed in the headcap, 
3) a wireless-to-wired bridge for bidirectional signal transfer, 
and 4) client software. Each headstage samples 32 channels, 
and each transceiver module connects to 4 headstages for a 
total of 128 channels handled by one transceiver. Up to four 
transceivers have been demonstrated to work simultaneously 
in rhesus monkeys. Recently, we found a solution to add 4 
more transceivers for a total of 1024 channels. The bidirec- 
tional wireless link serves to transmit neural information to the 
external units and to set the headstage/transceiver parameters 
by the operator. Spike sorting operations are performed by the 
transceiver, which reduces the amount of neural information 
transmitted wirelessly. The headstage and transceiver are 
powered by a lithium-ion cell, which is housed in the headcap 
and operates for 30 h continuously. This wireless, multichan- 
nel recording system is connected to a BMI suite. The system 
has been already used in a variety of BMI tasks, ranging from 
BMI control of a cart by a monkey housed in its home cage 
(711), to a monkey performing BMI whole-body navigation 
tasks while seated in a motorized wheelchair (656). Several 
multichannel, wireless recording systems have been developed 
by other groups, as well (81, 119, 135, 340, 429, 550, 557, 
602, 783). 


Vi. DECODING OF BRAIN SIGNALS 


A. Principles of Neural Decoding 


Modern real-time BMI computational algorithms, or de- 
coders, are employed for transforming neuronal activity 
into signals suitable for direct communication of the sub- 
ject’s brain with artificial actuators. Such decoders can em- 
ploy a large variety of statistical and machine-learning 
methods. BMI decoding algorithms belong to the class of 
multiple-input and multiple-output (MIMO) models (432), 
where multiple inputs are provided by the neural recording 
channels and multiple outputs correspond to the behavioral 
variables controlled by the BMI and/or signals for commu- 
nication with the external world. 


A decoder applies a transform algorithm to neuronal inputs to 
calculate the output variables. In many cases, the transform 
algorithm has many independent parameters that need to be 
mapped to a much smaller list of output variables. Setting the 
values of these parameters is called decoder training. There are 
different methods to train a decoder. The most traditional one 
requires sampling an initial segment of input data from which 
correlations between neuronal signals and behavioral vari- 
ables of interest are determined. The decoder performance is 
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evaluated using the comparison of the actual behavioral pa- 
rameters and the values derived by the decoder from the neu- 
ronal signals. After the decoder reaches high performance for 
the training segment, the BMI mode of operation can begin 
(851). At this point, the parameters needed to control an ex- 
ternal device are derived solely by real-time decoding of the 
incoming neural activity. 


The original BMI experiment performed by Carmena et al. 
(114) (FIGURE 5) can be used as an example of the tradi- 
tional training paradigm to create a BMI decoder. The ex- 
periment started with monkeys using one hand to operate a 
joystick linked to a robotic arm. The joystick movements 
were translated into the reaching movements performed by 
the robot. Additionally, monkeys activated the robot grip- 
per by squeezing the joystick handle. This hand-control 
phase lasted for ~15 min. Neuronal recordings obtained 
during this period provided the training data for a linear 
decoding algorithm, called the Wiener filter (see below), 
which represented the robotic arm kinematics and the grip- 
ping force as weighted sums of the neuronal discharge rates. 
Once the Wiener filter was trained, the operation was 
switched to BMI control, where the joystick was either elec- 
tronically disconnected from the robot or in some experi- 
ments physically removed from the setup. At this point, the 
monkeys used the Wiener filter’s outputs to directly control 
the robotic arm’s reaching and grasping movements. This 
type of BMI operation was called brain control. 


The shift from the training phase to brain control is often 
accompanied by changes in the patterns of overt movements 
produced by monkeys (114, 124, 463). Often, animals tend to 
diminish or eliminate their arm movements when operating a 
BMlin brain-control mode. Furthermore, the animals’ cortical 
neuronal firing patterns change significantly during the transi- 
tion from the training period to brain-control mode. These 
physiological changes are manifested as increases in correla- 
tion between cortical neurons, within and between cortical 
areas, (114, 583), and alterations in the directional tuning of 
individual neurons (283, 463). If these changes occur, the de- 
coder performance may deteriorate because it was trained un- 
der different behavioral conditions. To mitigate this problem, 
several strategies have been designed to adapt to these new 
behavioral and physiological conditions (114, 180, 283, 492, 
609, 794). For example, the initial training period can be elim- 
inated and the adaptation could begin from arbitrary decoder 
settings and continue throughout the experiment (283). Addi- 
tionally, the initial training can be conducted without any 
overt movements, using only passive observations by the sub- 
ject (377, 800, 827) and/or their mental imagery of move- 
ments (360, 362). 


As mentioned above, after the original introduction of lin- 
ear models for neural decoding in BMIs (374, 852), many 
other decoding algorithms have been proposed over the 
past decade. Indeed, describing a new BMI real-time com- 
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putational decoding strategy is the theme of a growing lit- 
erature in the field. Below we provide a brief summary of 
the main BMI decoders reported in the literature. 


B. Linear Decoders 


Linear decoders compute the output variables as weighted 
sums of the recorded neuronal rates (114, 794, 851, 852). 
Humphrey, Schmidt, and Thompson first proposed this 
idea in 1970 (374). They successfully demonstrated that 
movement parameters can be reconstructed from the re- 
cordings of the firing rates of multiple neurons using mul- 
tiple linear regression. Schmidt and his colleagues then ran 
this linear decoding in real time (702, 703). Georgopoulos 
was a notable proponent of the theory that such weighted 
summation constitutes a fundamental mechanism by which 
cortical neuronal populations represent motor variables 
(295, 297, 299). Georgopoulos’ theory expressed the con- 
tribution of each neuron to population encoding as a vector 
that pointed to the so-called preferred direction of that neu- 
ron (294, 710). The preferred direction was defined as the 
direction for which the neuronal firing rate was maximal. 
Next, the individual-neuron unit vectors, multiplied by the 
discharge rate of the corresponding neuron, were summed 
to form a population vector. The population vector turned 
out to be an excellent method to continuously extract the 
direction of arm movement from the activity of a popula- 
tion of M1 neurons. 


Since its introduction, the population vector approach to 
neural decoding has been very influential even though its 
original description was not based on simultaneous record- 
ings from many neurons. Instead, Georgopoulos and his 
colleagues (295, 299) employed traditional single-electrode 
neurophysiology to record froma single neuron or, rarely, a 
few neurons at a time. Artificial neuronal populations were 
assembled from sequentially recorded neurons on different 
days, and population vectors were calculated for those pop- 
ulations. Although Georgopoulos later employed a seven- 
electrode apparatus for his motor cortical recordings (36, 
173, 570), he did not record neuronal samples large enough 
to enable real-time decoding of arm movements using the 
population vector. 


In the classical Georgopoulos paradigm, the neuronal pre- 
ferred directions were derived from a behavioral paradigm 
called a center-out task. In this task, monkeys were required 
to perform arm reaching movements from an initial, central 
location, to a set of peripheral locations arranged in two- 
(294) or three-dimensional (299) space. Once the preferred 
directions were determined, the motor task could remain 
the same, or a more sophisticated task could be introduced. 
For example, Georgopoulos et al. (297) employed a cogni- 
tive task where monkeys had to perform a 90-degree mental 
rotation from the location of a visual stimulus to the motor 
target instructed by that stimulus. The population vector 
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proved to be informative of the representation of mental 
rotation by M1 neurons even though the neuronal preferred 
directions were derived from a simpler, center-out task. 


This early work on population-vector decoding contained 
an estimation of how many neurons would be needed to 
decrease noise in the extraction of arm kinematics from M1 
ensembles (295). The decoding noise was expressed as the 
variability of the calculated population vector direction for 
different realizations of single-trial ensemble firings. This 
parameter was then plotted against the population size. The 
curve showed an initial rapid decrease in variability as the 
population size increased, and then followed a much slower 
rate of decrease after the population reached the size of 150 
neurons. The slow decrease continued until all 475 neurons 
recorded in that study were included in the population. 
These estimations are consistent with our analysis of neu- 
ronal dropping curves constructed for simultaneously re- 
corded populations of cortical neurons (467, 583). 


Notwithstanding the elegance and theoretical importance 
of the population vector approach, this decoding method is 
suboptimal because the weights given to different neurons 
are chosen intuitively and without any provision for mini- 
mizing decoding errors. Additionally, the approach where 
the algorithm parameters are derived from one task (center- 
out task) and then applied to a different motor task (e.g., 
mental rotation) may result in additional errors because 
neuronal tuning properties could be different in the new 
task. Such a change in neuronal tuning is consistent with the 
context principle discussed above: under different condi- 
tions, cortical neurons tend to exhibit different activity pat- 
terns. 


Another decoding algorithm, the Wiener filter, has been 
successfully employed in various BMI studies to extract 
limb kinematics and other behavioral variables from neu- 
ronal population activity (114, 283, 284, 432, 468, 656, 
852). The Wiener filter is an optimal linear decoder set to 
minimize mean-square error (347, 485, 854). In a typical 
implementation, the Wiener filter output for time t is com- 
puted as a weighted sum of neuronal rates sampled at sev- 
eral time points, called taps or lags, preceding t. Ten taps 
with 100 ms spacing is a typical setting in monkey BMI 
experiments (114, 463). The filter weights are computed by 
applying matrix transformations to the training data that 
include the recordings of neuronal rates and behavioral 
variables of interest. 


The total number of Wiener filter weights depends on the 
population size and the number of taps. For neuronal pop- 
ulation of size N and number of taps T, the number of 
weights is equal to N multiplied by T. The number of 
weights, or free parameters, is referred to as the dimension- 
ality of the decoder. An excessive number of weights may be 
harmful for decoding accuracy because of the problem 
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known in many disciplines as overfitting (38, 338, 345) or 
the curse of dimensionality (821). An overfitted decoder 
may start fitting data to noise. Such fitting may appear to 
work perfectly for the training period, but the decoder 
would then fail when applied to a different data segment. 
To reduce overfitting, BMI decoding algorithms incorpo- 
rate such methods as regularization and dimensionality re- 
duction (432). 


Over the past 17 years, our laboratory has employed mul- 
tiple Wiener filters, running in parallel, for generating mul- 
tiple BMI outputs. For example, Carmena et al. (114) im- 
plemented two Wiener filters that generated x and y com- 
ponents of the robot velocity, and a third filter that 
generated the robot’s gripping force (FIGURE 5). The same 
approach was subsequently used to extract kinematics of 
lower limb motion during bipedal locomotion (261), de- 
code arm EMGs from cortical ensemble activity (694), de- 
code cortical representation of time intervals (468), and 
control whole-body navigation by monkeys seated in a mo- 
torized wheelchair (656). Overall, this is a powerful and 
easily tractable decoding method. 


C. Kalman Filter 


The Kalman filter (403, 404) is another popular algorithm 
for BMI decoding (432, 491, 620, 724, 869) that was pre- 
viously employed in numerous engineering applications 
(319). Like the Wiener filter, the Kalman filter takes multi- 
channel neuronal signals as inputs and returns predictions 
of behavioral variables as its output. The filter models neu- 
ronal inputs as observations and the behavioral variables as 
state variables. The state variables may include, for exam- 
ple, the position and velocity of the arm. 


The Kalman filter updates states in discrete time steps (usu- 
ally 50-100 ms). Each update consists of two calculations. 
The first calculation, called predict step, provides an esti- 
mate of the next state from the previous state based on a 
state transition model, also called a movement model in 
BMI applications. For a robotic arm movement, for exam- 
ple, the next state can be estimated based on the previous 
position and velocity. The second calculation, called update 
step, performs an adjustment of this estimated state using 
the observed neuronal rates. This calculation employs an 
observation model, or neuronal tuning model, that repre- 
sents neuronal rates as a function of state variables. The 
directional tuning curve is an example of such an observa- 
tion model as it describes neuronal firing rates as a function 
of movement direction. During the update step, the obser- 
vation model converts the estimation of state into an esti- 
mation of expected neuronal rates. Then, the estimated neu- 
ronal rates are compared with the actual recorded neuronal 
rates, and the state is adjusted to accommodate this differ- 
ence. This final computation of the state forms the filter 
output. 
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While several groups have reported that the Kalman filter 
outperforms other linear decoders (724, 869), there is also a 
report of a very similar performance of these methods for a 
different dataset (432). Thus, currently, there is no general 
recommendation on which algorithm should be chosen for 
BMI decoding. Instead, choices should be made after con- 
crete conditions and requirements are examined. 


A few years ago, our laboratory introduced an improve- 
ment to the Kalman-based BMI decoder, based on the un- 
scented Kalman filter (UKF) (491). This algorithm was de- 
signed for handling nonlinear observation and state transi- 
tion models (392). The UKF is relevant for neural decoding 
because there are nonlinearities in the relationship between 
neuronal rates and limb kinematics, such as the one relating 
neuronal tuning to speed in addition to velocity (492, 554). 
In our implementation, an mth order UKF included two 
novel features: 1) a nonlinear model of neural tuning which 
incorporated absolute values of velocity and radius, and 2) 
an addition of 1-1 recent states to the state variables. This 
new decoder outperformed the classical Kalman filter and 
the Wiener filter when applied to the data from a center-out 
task and a target tracking task. Moreover, the UKF outper- 
formed the other decoders when used for real-time BMI 
control of movements of a computer cursor. Following this 
study, we have used the UKF in a number of BMI studies, 
including a BMI for controlling one avatar arm (598) or 
two avatar arms simultaneously (377). 


D. Point-Process Models 


Point-process models of neuronal spiking activity employ a 
likelihood function to describe the probability of a neuron 
to produce a spike. The likelihood function depends on 
such parameters as the neuronal spiking history, activity of 
the other neurons in the population, external stimuli, and 
behaviors (144, 809). An analog of the Kalman filter can be 
formulated using a point-process model of the observation 
state (97, 222, 490). Indeed, both the Kalman filter and 
point-process decoders utilize the concept of state to de- 
scribe both the decoded variables and neural activity. State 
transitions are described by statistical models (450). 


Although computationally demanding, point-process de- 
coders in certain cases offer better temporal resolution com- 
pared with neural decoders that decrease the resolution by 
down sampling the neuronal activity into bins (with a typ- 
ical bin width of 50-100 ms) (461, 728-730, 807, 816, 
872). 


E. Artificial Neural Networks 


Artificial neural networks (ANNs) were introduced as de- 
coders in the early BMI studies of the late 1990s (124, 852). 
Since then many ANNs have been used in both invasive and 
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noninvasive BMIs, including a multilayer perceptron (22, 
161, 369, 428, 431, 691), adaptive logic network (444, 
445), tree-based neural network (380), and learning-vector- 
quantization (402, 460, 632). 


Krishna Shenoy and his colleagues (776) have introduced a 
decoding method based on a dynamical ANN, called recur- 
rent neural network (RNN). In this algorithm, neuronal 
activity is considered as a function of its history in addition 
to being related to motor parameters. In Shenoy’s study, the 
RNN continuously decoded the kinematics of center-out 
arm reaching movements from monkey M1 activity. This 
decoding scheme outperformed the velocity Kalman filter in 
the same task. 


F. Adaptive Decoders 


Adaptive decoders make adjustments that improve BMI 
performance while the subject continuously operates a 
BMI. While the decoder adapts, changes occur in the brain 
itself owing to neuronal plasticity. 


The first adaptive decoder introduced in BMI literature was 
a coadaptive algorithm implemented to improve real-time 
conversion of the activity of monkey M1 neurons into the 
three-dimensional center-out movements of a cursor (794). 
The cursor position was generated by a population-vector 
decoder. The coadaptive algorithm adjusted the population 
vector weights after each trial to bring the BMI-generated 
trajectories closer to the ideal trajectories connecting the 
cursor’s initial position to the target. Following that study, 
many adapting algorithms have been developed. 


Liet al. (492) developed a Bayesian regression self-training 
algorithm that updated the settings of a UKF. That BMI 
utilized neuronal ensemble recordings from multiple corti- 
cal areas in rhesus monkeys to control two-dimensional 
cursor movements. The adaptive algorithm monitored the 
decoder output and periodically updated the UKF neuronal 
tuning model based on the detected changes. The updates 
were performed using Bayesian linear regression. The on- 
line performance of this algorithm was tested in 11 experi- 
mental sessions that spanned 29 days. The initial parame- 
ters of the decoder were trained on the first day of record- 
ings, and the evolution of these parameters was performed 
by the adapting algorithm without any retraining sessions. 
The adaptive decoder secured stable BMI performance, 
whereas the performance deteriorated if the unchanged ini- 
tial decoder was used. Dangi et al. (181) used a similar 
two-dimensional reaching task as a test bed for developing 
a general framework for selecting parameters to adapt and 
the adaptation timescale. They also developed tools that 
evaluated convergence properties of adaptive algorithms. 


Several studies introduced supervised learning algorithms 
that used the information about target location to adap- 
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tively improve BMI performance. Kowalski et al. (448) em- 
ployed a naive adaptive BMI for this purpose. The system 
jointly analyzed neuronal patterns and user intent of reach- 
ing to the target. Shanechi and Carmena (728) used a sim- 
ilar idea of analyzing the user intent. They developed an 
adaptive optimal feedback-controlled point process de- 
coder that derived subjects’ intentions from the relative po- 
sition of the cursor and target. This adaptive scheme im- 
proved the performance even when the decoder’s initial 
parameters were set to arbitrary numbers. Along similar 
lines, Suminski et al. (774) developed a kinetic decoder that 
continuously adapted joint torques based on the discrep- 
ancy between the target location and hand position. The 
updates were performed using gradient descent. 


Justin Sanchez and his colleagues (197) employed reinforce- 
ment learning as a BMI adapting algorithm. In this ap- 
proach, actions that maximized the reward were selected 
through trial and error. Trial outcome (i.e., presence or 
absence of reward) served as a scalar signal that was utilized 
for parameter updates at the end of each trial. Two studies 
from this group (520, 640) showed that actor-critic rein- 
forcement learning could quickly recover decoding accu- 
racy when neural inputs were lost or shuffled. Moreover, 
the same group showed that the reinforcement signal for 
such learning could be derived from the recordings in nu- 
cleus accumbens (520). 


G. Discrete Classifiers 


Discrete classifiers convert neuronal activity into discrete 
choices. These decoders are commonly used in noninvasive 
BMIs, where subjects generate a limited number of outputs, 
often just two (333, 569, 639). Discrete classifiers have been 
used in some intracranial BMIs, as well (343, 692), includ- 
ing systems that combined both discrete and continuous 
decoders (261, 816). 


The mathematical algorithms for discrete classification in- 
clude linear discriminant analysis (LDA) (259, 288, 629), 
support vector machine (56, 288, 780), ANNs (359), mul- 
tilayer perceptron (43, 75), hidden Markov models (603, 
655), k nearest neighbors classifier (177), and nonlinear 
Bayesian classifiers (191). 


Vil. MOTOR CONTROL WITH 
INTRACRANIAL BMis 


A. Theories of Motor Control as a 
Foundation for Motor BMis 


Motor BMIs extract motor commands from a sample of 
neuronal activity and send this control information to ex- 
ternal devices that execute the movements imagined by the 
operator. At the basic science level, these systems are in- 
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tended to investigate the physiological properties of motor 
circuits, theories on neuronal encoding, and the impact of 
learning and plasticity on neuronal ensembles. From a clin- 
ical point of view, BMIs primarily aim to restore crucial 
motor behaviors, such as arm movements or locomotion, to 
patients suffering from devastating levels of body paralysis, 
as a result of brain trauma or degenerative neurological 
diseases. 


The design of existing motor BMIs, in many ways, matches 
current theories of the motor system layout and operation. 
The first design issue is where in the brain to record neural 
activity that would be converted into motor commands to 
an external device. This issue is closely related to the assess- 
ment of functional roles of different brain areas and the 
types of neural processing they perform. Since the motor 
system in primates is defined by a highly interconnected 
network of cortical, subcortical, and spinal structures, in 
general, there are many brain areas that could provide in- 
puts for motor BMIs. Classically, the motor system is de- 
scribed as being formed by a hierarchy, in which cortical 
motor areas are presumed to handle advanced or higher 
order functions, for example, dexterous hand movements. 
Meanwhile, lower-order subcortical areas are presumed to 
manage less complex, automated motor acts. In this motor 
hierarchy, the spinal cord has been traditionally believed to 
handle low-order functions, such as reflexes (733) and cen- 
tral pattern generators (CPGs) (328). Reflexes are auto- 
mated, and often unconscious motor responses to sensory 
stimuli. In contrast, voluntary movements are prepared and 
executed under cortical control. They may be related to 
external stimuli, but may also originate in the mind rather 
than being caused by sensory inputs. While there are merits 
to the classification of motor activities into less advanced, 
automated responses and more advanced, voluntary ac- 
tions (65), our work has repeatedly shown that there is a 
constant flow of information between cortical, subcortical, 
and spinal structures during the execution of motor behav- 
iors (576, 578, 617, 894). In this distributed view of the 
motor system, there is no clear-cut separation between 
high-order and low-order processing. In support of our 
view, practically any motor task involves a mixture of vol- 
untary and reflex activities (158). 


Several theories of motor control have influenced the design 
and experimentation with BMI. For instance, the concept of 
body schema, a quite important concept for modern BMI 
research as we discuss below, was originally proposed by 
Head and Holmes one century ago (351). According to 
Head and Holmes’ original formulation, the brain creates 
an internal model of the body, the body schema, which 
governs motor activities and perceptions. The body schema 
is constantly updated by streams of sensory information. 
With the emergence of BMI research, the concept of body 
schema was not only investigated but acquired a complete 
new angle; one in which artificial prosthetic limbs, con- 


trolled directly by the patient’s own brain activity during 
the utilization of a BMI, are believed to be assimilated into 
the body schema as extensions of the subject’s biological 
body, through the process of plasticity triggered by BMI 
long-term usage (466). 


Modern theories of motor control are rooted in the ideas of 
Head and Holmes and, as such, have become the subject of 
investigation by BMI research. The internal model theory 
(310, 415, 459, 866) describes the motor system as being 
defined by two components: the controlled object (e.g., a 
body part or the entire body) and the controller. The con- 
troller uses an internal model to program future motor 
states. When the object movement is executed, the control- 
ler compares the expected state with the actual sensory 
feedback from the controlled object. If a discrepancy be- 
tween the expected and actual state is detected, the control- 
ler issues a correction command. It has been suggested that 
to be efficient, BMIs should perform similar forward plan- 
ning based on an internal model (175, 309). 


Another popular motor control theory, Feldman’s equilib- 
rium point theory (249, 250) suggests a possible neural 
mechanism to implement the controller. In this theory, 
higher-order motor centers manage the position of an equi- 
librium point for the limb, and the limb is brought to the 
equilibrium point by a spinal servo mechanism. BMIs with 
a similar separation between the higher-order and low-or- 
der controls have been proposed. In this design, called 
shared-control BMI, high-order motor commands are ex- 
tracted from cortical activity, whereas the low-order execu- 
tion is delegated to a robotic controller, which handles the 
“equilibrium point” using Feldman’s terminology (427, 
634). 


Optimal feedback control is yet another popular motor 
control theory (263, 801, 802). This theory describes an 
optimal strategy for using multiple biomechanical degrees 
of freedom to achieve the goal of a motor action. The strat- 
egy is based on stochastic optimal feedback control that 
corrects deviations in the degrees of freedom that define 
task goals, while allowing variability in task-irrelevant di- 
mensions. The theory explains such phenomena as motor 
variability, error corrections, and motor synergies. Several 
BMI decoders that implement optimal feedback control 
have been proposed (59, 729, 730). These BMIs estimate 
the performance error by comparing the current location of 
an actuator with the planned trajectory estimated from the 
neuronal signals. A correction is then issued in the appro- 
priate dimensions. 


B. BMI Control of Virtual and Robotic Limbs 


Motor BMIs that enable upper limb functionality, for ex- 
ample, a BMI for arm reaching and grasping (114) (FIGURE 
5), have received particular attention because of the obvious 
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key importance of arm movements in our daily life. The first 
BMI of this type operated in an open-loop mode, i.e., with- 
out any sensory feedback from the BMI-controlled actuator 
(852). In that experiment, while New World monkeys ma- 
nipulated a joystick, their cortical activity was decoded and 
converted into the movements of a robotic arm using an 
Internet protocol. The monkeys did not see the robot, which 
was located in a different state many hundreds of miles 
from the animal. All subsequent BMI demonstrations uti- 
lized a robotic arm operated in a closed-loop mode (114, 
463, 794, 817), where monkeys received visual feedback of 
the robotic arm movements and could correct their perfor- 
mance errors. 


The main components of a motor BMI that controls a ro- 
botic arm are featured in the system developed in our lab- 
oratory in 2003 (114) (FIGURE 5). Here, rhesus monkeys 
learned to control reaching and grasping movements per- 
formed by a robotic arm by using only the combined elec- 
trical activity of cortical ensembles recorded with multi- 
electrode arrays, built from flexible Teflon-coated mi- 
crowires chronically implanted in multiple cortical areas, 
including M1, $1, PMd, supplementary motor area (SMA), 
and posterior parietal cortex (PPC). These cortical areas 
were chosen because they belong to the frontal-parietal cor- 
tical circuitry that controls goal-directed arm and hand 
movements (20, 399, 470). 


Two monkeys initially executed a motor task, placing a 
computer cursor to the center of a moving circle that served 
as a target, manually (FIGURE 58). To do this, monkeys 
grasped a joystick and shifted it in different directions; the 
joystick position was translated into the cursor position on 
the screen. Later stages of this behavioral task also required 
that the monkeys apply gripping force to the joystick han- 
dle, at the end of the reaching movement, so that they could 
imitate grasping the virtual target. Next, monkeys learned 
to control the reach and grasp movements of the robotic 
arm equipped with a gripper. Since the joystick was con- 
nected to the robot arm, when the monkey moved the joy- 
stick and applied hand gripping force to it, the robot arm 
and gripper reproduced these movements. The visual feed- 
back of the robot movements was delivered to the computer 
screen, where the robot position was represented by a cir- 
cular cursor and the gripping force was represented by the 
cursor diameter. The virtual targets that the robot had to 
reach and grab were represented by circles of varying diam- 
eters. To win a fruit juice reward, monkeys had to move the 
robot, place its gripper over the virtual target, and then 
produce the correct level of gripping force, to match the 
cursor diameter with the diameter of the target. 


While monkeys practiced these reach and grasp task, the 
firing rates of ~100 cortical neurons, distributed across the 
cortical areas mentioned above, were fed into multiple Wie- 
ner filter algorithms so that multiple parameters could be 
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generated continuously to control the reach and grasp 
movements of the robotic arm. In brain-control mode, the 
joystick was electrically disconnected from the robot and 
the outputs of the Wiener filters defined the robot move- 
ments. The monkeys continued to manipulate the joystick 
with their hands although it was disconnected from the 
system. After the monkeys perfected this brain control as- 
sisted by the joystick movements, the joystick was removed 
from the setup. At this stage, to receive its fruit juice reward, 
the monkey could no longer rely on the well-trained joystick 
task. Instead, they had to learn to control the robot with 
their own cortical activity without assisting themselves with 
arm movements. The performance errors were initially 
high, but then decreased as the monkeys practiced in the 
brain control without hand movements. Learning to con- 
trol this BMI was accompanied by a transient increase in 
correlation between the simultaneously recorded cortical 
neurons (114, 583), within and between cortical areas, and 
by changes in neuronal tuning to the robot arm movements 
(463). 


Several other groups developed BMIs for arm reaching, as 
well. Schwartz and his colleagues have explored the possi- 
bility of performing BMI control over reaching in three 
dimensions (3D). In one study, they trained monkeys to 
wear stereoscopic goggles that displayed 3D movements of 
a cursor (794). In the beginning of each trial, the cursor was 
positioned in the center of this virtual reality display. A 
spherical target then appeared at a random 3D location, 
and the monkeys acquired it with the cursor to receive a 
reward. During the manual control mode, monkeys waved 
their hands in the air to move the cursor. The hand move- 
ments were monitored by a video tracking system. During 
the brain control mode, the cursor was moved by cortical 
activity processed by a decoding algorithm. Initially, the 
researchers attempted to train a population vector decoder 
using the manual performance data, and then use that de- 
coder for brain control. However, after realizing that this 
control was not sufficiently accurate, they sought an adap- 
tive algorithm that would improve the performance. Their 
adaptive decoder, called coadaptive movement prediction 
algorithm, adjusted the decoder parameters so that the tra- 
jectories generated by the BMI were brought closer to the 
ideal linear trajectories connecting the initial central posi- 
tion and the target. 


Building on these results, the Schwartz laboratory devel- 
oped a BMI for monkey self-feeding (817). For this purpose, 
they used a robotic arm equipped with a gripper that picked 
a piece of food and brought it to the monkey’s mouth. These 
experimental settings resembled the previous study of Leb- 
edev and Wise where a robotic manipulator brought food 
to monkeys (471). In Schwartz’s study, the robot was con- 
trolled by a linear decoder that transformed cortical neuro- 
nal population activity into the velocity of the robot’s end 
point. The gripper’s opening and closing was commanded 
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by cortical activity, as well. The authors reported a curious 
type of learning during these BMI operations: monkeys 
learned to start opening the gripper before it reached the 
target. They could do this without risking dropping a piece 
of food because marshmallows that stuck to the gripper 
were used as rewards. Although the monkeys possibly fo- 
cused less on controlling the feeder because of the sticky 
rewards, this observation illustrates that BMI control, like 
normal motor control, can undergo adaptation. 


Following these demonstrations of unimanual BMI control, 
the Nicolelis laboratory took the next logical step in this 
research by demonstrating a BMI that controlled two arti- 
ficial arms simultaneously (377). In that study, rhesus mon- 
keys viewed two virtual arms on a computer screen, and 
commanded their reaching movements using a cortical BMI 
that utilized the extracellular activity of ~400 neurons sam- 
pled in multiple areas of both hemispheres, including M1, 
$1, PMd, and SMA. Cortical ensemble activity was con- 
verted into bimanual movements using a UKF that treated 
the kinematic parameters of both arms as parts of the same 
state model. The decoder was trained either using a joystick 
task where monkeys moved the virtual arms with two joy- 
sticks or through a passive observation task that required 
monkeys to watch the virtual arms move on the screen. 
Eventually, the monkeys were able to control the virtual 
arms by their cortical activity without moving their own 
arms. This learning was accompanied by widespread corti- 
cal plasticity that manifested itself by an increase in cortical 
responses to the observation of virtual arm movements and 
by changes in pairwise correlations between neurons. 


John Donoghue’s group at Brown conducted several BMI 
studies in implanted humans. In these studies, paralyzed 
patients were implanted with the Utah array in the M1. The 
patients learned BMI control of a screen cursor (361) or a 
robotic arm (361). One of the patients learned to grasp a 
coffee bottle with a robotic hand and, somewhat slowly 
(more than 1 min per trial), bring it to her mouth. The 
slowness of operation was possibly related to a deteriorated 
quality of neuronal recordings. The study did not document 
the number of neurons performing the control. Instead, it 
reported that neuronal electrical signals were picked up by 
96 recording channels. A simple threshold crossing proce- 
dure was used to detect multiunit activity. The decoding 
was performed using a Kalman filter that was initially 
trained to predict robot hand displacements as the patients 
observed the movements of the robotic arm and imagined 
themselves controlling those movements. The Kalman filter 
decoder was iteratively adjusted during the phase of BMI 
control. To ease the learning, the patient’s performance was 
corrected by computer commands that brought the robot 
arm closer to the optimal trajectory. This procedure, called 
“error attenuation” consisted of decreasing the robot 
movement commands orthogonal to the trajectory connect- 
ing the robot to the target. The contribution from the error 


attenuation routine was gradually decreased and eventually 
removed. The robot hand state was controlled using an 
LDA classifier that, similarly to the velocity decoder, was 
trained using observations of the robot movements com- 
bined with motor imagery. The drinking task was further 
assisted by a preprogrammed sequence of actions. First, the 
LDA classifier commanded an automated impedance-con- 
trolled grasping and lifting the bottle. Second, the same 
classifier stopped the movements of the robot arm and pr- 
onated the robot wrist to point the bottle toward the pa- 
tient. Third, the robot wrist was brought to its initial posi- 
tion and arm movements allowed, and fourth, the bottle 
lowered to the table and released. Such a mode of opera- 
tion, where control functions are distributed between a BMI 
operator and the robotic controller, is referred to in the 
literature as shared control (223, 379, 427). 


While the experiments of Donoghue and his colleagues have 
demonstrated that patients with upper-limb paralysis can 
employ their cortical activity, recorded from the arm and 
hand representations in MI, to control the reaching and 
grasping movements performed by a robotic arm, several 
key questions remain regarding the nature of this control. 
The videos from their experiments show that the subjects 
could move their heads. In some of the trials, they clearly 
tracked the robot displacement with head movements. Arm 
movements accompanying the grasp command to the robot 
are also noticeable in the videos. These observations suggest 
that cortical neuronal activity related to head movements, 
which, in these patients, likely expanded beyond the origi- 
nal head representation before the trauma or disease, could 
have been involved in controlling the robot arm, in addition 
to the newly created cortical representation of the robot. 


Overall, the role of assistive overt behaviors in BMI control, 
i.e., movements of body parts and the eyes that could be 
used to generate neural inputs for a BMI, is often neglected 
or downplayed in the literature. Certainly, this topic will 
require more scrutiny in the future. Historically, neuro- 
physiological experiments strived for maximal control of 
unwanted overt behaviors. For example, neurophysiolo- 
gists have developed an instructed delay task where an an- 
imal is not allowed to produce any motor output while pre- 
paring a movement (860). However, even if an instructed- 
delay task is well-learned, and no overt movement occurs, 
motor preparation still causes activation of spinal circuits 
involved in low-level motor control (653). Therefore, even a 
very clean BMI experiment, where the subject does not 
move the limbs or eyes, may involve activation of both 
higher-order brain areas that drive the BMI, and low-order 
subcortical and spinal regions. This is not a problem for 
practical BMI implementations, but rather an issue that 
needs to be better understood. For practical BMIs, even the 
presence of overt behaviors can be useful because they could 
improve the subject’s performance. Indeed, BMIs that mix 
several brain-derived signals with the signals representing 
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overt behaviors, for example, eye movements or EMGs, are 
called hybrid BMIs (337, 387, 476, 626, 833). 


In their clinical trials, Schwartz and his colleagues further 
improved the accuracy BMI control exerted by a paralyzed 
human. They recorded from close to 200 neurons in an 
individual with tetraplegia implanted with multielectrode 
arrays in the motor cortex (152). As predicted in the early 
2000s by our lab (114, 852), the increase in the number of 
simultaneously recorded neurons led to an improvement in 
the BMI performance. The subject gained control of an 
anthropomorphic robotic arm that performed skillful and 
coordinated reaching and grasping movements, like reach- 
ing to a knob and then turning it clockwise or counterclock- 
wise. Like in Donoghue’s experiments, many of these ex- 
periments utilized assisted BMI control, where the subjects’ 
errors were corrected by the controller to facilitate learning. 
The subjects were eventually able to operate without that 
assistance. 


Altogether, these clinical studies demonstrated the feasibil- 
ity of implementing cortically controlled BMIs to reproduce 
upper limb movements. They also exposed a number of 
issues that preclude immediate translation of these systems 
into the clinical arena (466). One issue is the requirement 
for practical neural prostheses to be fully implantable. 
Wired implants are suitable for animal experiments and 
short-term clinical trials, but not for devices aimed at serv- 
ing as long-term clinical solutions. In a practical clinical 
system, implanted electrodes and preamplifiers should be 
fully contained under the scalp while wireless technology is 
used to transfer large-scale recorded neural signals. Further- 
more, implant biocompatibility remains a problematic is- 
sue. The utilization of the Utah probe, in both monkeys and 
human subjects, has repeatedly shown that the quality of 
neural recordings tends to deteriorate with time due to elec- 
trode encapsulation and neuronal tissue loss, likely as a 
result of the tissue injury caused by the electrodes. Finally, 
there are many challenges for real-time decoding algo- 
rithms, which currently are limited to small sets of motor 
behaviors. 


C. BMI for Walking 


During the last two decades of explosive BMI development, 
research focused mostly on controlling neuroprosthetic de- 
vices that mimic upper limb functions. Yet, tens of millions 
of people worldwide suffer from paralysis of the lower 
limbs as a result of trauma to the spinal cord or neurode- 
generative diseases that affect the peripheral nervous sys- 
tem. Additionally, there are millions of lower limb ampu- 
tees and patients who suffer from neurological disorders 
that affect gait, such as Parkinson’s disease. 


A cortically driven BMI for decoding of bipedal walking 
was first developed by our laboratory (261) (FIGURE 12). In 
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these experiments, two macaque monkeys were trained to 
walk bipedally on a treadmill while holding a bar with their 
hands to assist balance. Next, the monkeys were implanted 
with multielectrode arrays placed in the regions of M1 and 
S1 representing the lower limbs. The neuronal ensemble 
recordings conducted with these implants showed that, 
while monkeys walked on the treadmill, cortical neuronal 
discharges were correlated with the stepping movements. 
Owing to these correlations, the Nicolelis lab researchers 
could extract multiple lower limb kinematic parameters 
from the cortical recordings. Multiple Wiener filters were 
used for that purpose, which extracted 3D position of the 
hip, knee, and ankle joints, as well as the EMGs of leg 
muscles. The decoding reconstructed movement patterns of 
both forward and backward walking. 


In a second series of experiments, using a custom-designed 
internet connection, the Nicolelis group transmitted the 
output of their BMI to a humanoid robot built by Gordon 
Cheng and Mitsuo Kawato at The Advanced Telecommu- 
nications Research (ATR) Institute in Kyoto, Japan (133). 
The humanoid robot received continuous signals from the 
BMI through an optimized internet link that minimized the 
transmission delay. An image of the walking robot was 
projected to the screen mounted in front of the monkey. 
Initially, the robot was suspended over a treadmill. In later 
experiments, monkey cortical activity was employed to in- 
duce controlled bipedal walking of the same robot on the 
floor (414). 


After this study, decoding of kinematics of monkey quadru- 
pedal walking from cortical activity was demonstrated by 
other groups, again with good precision (267, 268, 711). 
Additionally, leg EMGs during standing and squatting were 
extracted from monkey M1 activity (889). 


For the case of quadrupedal locomotion, Capogrosso et al. 
(112) recently reported a “brain-spine interface” that alle- 
viated gait deficits in rhesus macaques with unilateral spinal 
cord injuries. They implanted rhesus monkeys with multi- 
electrode arrays placed in the leg area of M1, contralateral 
to the subsequent SCI site. Electrical stimulation was ap- 
plied epidurally to dorsal roots to produce extensions and 
flexions of the leg weakened by the SCI. Monkeys learned to 
volitionally control the paralyzed leg using the interface 
that converted cortical neuronal activity into the spinal 
stimulation patterns. The authors argued that, because they 
recorded in M1 representation of the affected leg, the ex- 
tracted motor commands represented intentions to move 
that leg. A careful examination of their methodology, how- 
ever, raises several pivotal concerns. First, they employed a 
decoder training procedure that relied on the presence of 
overt movements in the affected leg that exhibited “residual 
hip or knee oscillations.” Clearly, this could be related to 
the mechanical perturbations caused by the movements of 
the intact limbs, rather than voluntary attempts to execute 
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steps with the paralyzed leg. Second, the neuronal modula- 
tions in the M1 ipsilateral to the lesion could represent the 
movements of the intact limbs that could move normally. 
Such representation could occur as the result of cortical 
plasticity following the SCI and maintained by cortico-cor- 
tical connections (393). Thus factors different from the 
monkey’s true intention to move the paralyzed leg could 
underlie the cortical modulations that triggered electrical 
stimulation of the dorsal roots and evoked the artificial 
steps. 


In addition to assisting disabled people to regain the ability 
to walk, BMIs can be employed as a rehabilitation method 
(205). This latter approach, which has employed mainly 
noninvasive BMIs, will be discussed below. 


D. BMI for Whole Body Navigation 


Currently, the wheelchair is the main assistive device that 
enables navigation to people suffering from paralysis. Our 
laboratory also pioneered an intracranial BMI for wheel- 
chair control (656). In this study, two rhesus monkeys were 
trained to control a robotic wheelchair, while being seated 
on top of it, by the activity of their cortical neuronal ensem- 
bles. Monkeys were chronically implanted with microelec- 
trode arrays in multiple areas of both hemispheres. Neuro- 
nal ensemble activity in these areas was recorded using our 
wireless recording system (711). Wiener filters were used as 
decoders. Each experimental session started with a decoder 
training session, where the robotic wheelchair was driven 
by the computer; monkeys remained passive observers of 
these movements. During this passive navigation, two Wie- 
ner filters were trained to extract wheelchair kinematics 
from cortical activity. Such decoding was possible because 
cortical neurons were tuned to the wheelchair movements. 
One Wiener filter extracted translational velocity of the 
wheelchair (movements forward and backwards), whereas 
the other extracted rotational velocity (leftward and right- 
ward rotations). 


Following the training session, the mode of operation was 
switched to brain control, where the monkeys’ cortical ac- 
tivity was now mapped into the wheelchair’s translation 
and rotation velocities. The behavioral task consisted of 
driving the wheelchair toward a food dispenser that deliv- 
ered grapes as a reward. As the monkeys trained, their abil- 
ity to navigate the wheelchair with cortical signals im- 
proved. Additionally, performance on the wheelchair navi- 
gation task resulted in the emergence of a representation of 
the distance to reward location, a tuning property that re- 
sembled hippocampal place cells, in the primary motor and 
somatosensory cortical areas. This representation was to- 
tally unrelated to the settings of the decoder. 


While our BMI converted M1 and S1 activity directly into 
whole body navigation commands, without the need for 
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any intermediary overt behaviors, an alternative approach 
to enable such navigation is to have monkeys steer a mo- 
torized wheelchair with a joystick. Recently, it has been 
shown that monkeys can perform such steering to navigate 
a complex maze (226). Moreover, one study has shown a 
transition from joystick control to BMI control of a wheel- 
chair (494). In that study, neuronal ensemble activity was 
recorded using cortical arrays implanted in the arm repre- 
sentation of M1. Monkeys were initially trained to steer the 
wheelchair with a joystick. While they did so, a decoder was 
trained to classify the joystick steering commands based on 
M1 activity. Next, the mode of operation was switched to 
brain control, where the steering command was derived 
from cortical activity. Finally, the authors demonstrated 
that, like in our study, the decoder could be trained without 
the joystick movements. Curiously, activity patterns of 
some M1 neurons changed dramatically after the mode of 
operation was switched from joystick control to brain con- 
trol. Using the joystick in the context of BMI control of 
whole body navigation somewhat resembles the previous 
implementations of BMIs for arm reaching (114, 377). 
However, an important difference is that subjects have to 
learn a spatial transformation from the arm to the joystick 
movements (757). 


Overall, these studies have demonstrated that intracranial 
BMIs could drive a prosthetic device that enabled whole 
body mobility. Such a device could be used to restore mo- 
bility to severely paralyzed patients in the future. 


E. BMis That Utilize FES 


FES of peripheral nerves is a promising approach to restore 
motor functions to paralyzed subjects. FES-based BMIs aim 
to use the subject’s own brain activity to control the delivery 
of electrical stimulation to his/her own muscles that would 
then move their limbs. Over the past decade and a half, 
some progress has been reported with such BMIs. 


The initial evidence of the feasibility of BMIs that mimic 
muscle activity was provided by the demonstrations that 
EMGs of arm (558, 642, 694) and leg (261) muscles could 
be extracted from the activity of cortical neuronal popula- 
tions. Additionally, studies in healthy human subjects 
showed that a multichannel FES could produce near-nor- 
mal hand movement patterns (390, 713). 


The first demonstration of a BMI with FES output was done 
by Pfurtscheller and colleagues who aided a tetraplegic pa- 
tient with a FES device attached to his forearm (630, 633). 
The FES was controlled by bursts of cortical beta activity 
(18-25 Hz) recorded by EEG electrodes placed over the 
patient’s sensorimotor cortex while he tried to imagine 
moving his foot. After some practice, the subject learned to 
grasp objects using this device. 
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Several demonstrations of BMI-controlled FES have been 
accomplished in monkey studies. Eberhard Fetz’ group tem- 
porarily paralyzed monkeys’ hands with an anesthetic 
blockade (556) and then employed the firing rate of neurons 
located in the primary motor cortex (M1) to control an FES 
device that evoked wrist torques. Visual feedback of the 
torque was provided by a screen cursor. Monkeys success- 
fully learned to control this BMI. Moreover, M1 neurons, 
which were initially poorly associated with hand move- 
ments in a manual task, later on developed task-related 
modulations during the BMI control. 


Lee Miller and colleagues also demonstrated brain control 
with an FES device (225, 641). Their FES system was con- 
trolled by populations of ~100 motor cortical neurons re- 
corded with chronically implanted microelectrode arrays in 
the monkey M1. Hand paralysis in these animals was in- 
duced by an anesthetic block of the median and ulnar nerves 
at the elbow level. After the voluntary motor control of the 
hand was extinguished, the researchers activated forearm 
muscles with FES driven by the M1 signals. Monkeys were 
able to perform object grasping with this neural prosthesis. 


Bouton et al. (83) demonstrated a BMI with FES in a para- 
lyzed human with an intracranial multielectrode implant. 
The study subject suffered from a C5/C6 complete, non- 
spastic quadriplegia, resulting from a diving accident. A 
Utah array was implanted in the hand area of M1, which 
allowed recordings from up to 50 single units simultane- 
ously. The recordings continued for 350 days, and 33 units 
were isolated by the end of the study. During the training 
session, the subject attempted to produce six wrist and hand 
movements. These movements were impaired by the paral- 
ysis but could be evoked by FES. The FES was delivered 
using a 130-electrode array of surface electrodes embedded 
ina sleeve that was wrapped around the forearm. Neuronal 
population activity was converted into FES patterns using 
multiple simultaneous neural decoders based on a nonlinear 
kernel method with a non-smooth support vector machine. 
During the brain control mode of operation, the subject was 
required to generate hand movement that matched the cue 
shown on a computer screen. Following training, he man- 
aged to perform up to 70% of trials correctly. 


Notwithstanding the success of these demonstrations, using 
FES to restore movements meets a number of difficulties, 
such as muscle fatigue (224, 305, 796) and difficulties in 
achieving good accuracy of evoked movements without 
sensory feedback of force and position (25, 385, 818). 


F. Neuronal Plasticity in Motor BMIs 


Subjects usually experience difficulties when they are first 
introduced to brain-control mode of BMI operation. Yet, 
over time, they improve their performance with continuous 
practice. Such improvements have similar mechanisms as 


Physiol Rev - VOL 97 - APRIL 2017 - www.prv.org 


learning of new motor skills (1, 69, 212, 213, 356, 439, 
460, 547, 726). As such, many authors have proposed that 
neuronal plasticity is essential for BMIs to work properly in 
both animal and human subjects (114, 170, 195, 204, 324, 
463, 466, 583, 612). As a matter of fact, some authors have 
gone as far as to implicate cortical plasticity triggered by 
BMI operations as the key mechanism through which sub- 
jects could assimilate prosthetic limbs or even other actua- 
tors, such as virtual limbs, as extensions of the subject’s 
body schema created by the brain (466, 586, 738). 


In general, BMI control of an artificial actuator has much in 
common with the neurophysiological mechanisms involved 
in learning to use and become proficient in tool handling, 
operations known to evoke brain plasticity (67, 195, 378, 
524, 525). This likeness can be easily verified by reviewing 
the experiments conducted by Atsushi Iriki’s laboratory. In 
their fundamental experiments on primate tool usage, Iriki 
et al. (378) trained macaque monkeys to reach toward dis- 
tant objects, which could not be accomplished by using 
their arms alone, by utilizing an external tool: a rake. Before 
monkeys could use the rake, the researchers measured the 
receptive fields of multimodal neurons in the posterior pa- 
rietal cortex. Prior to the use of the artificial tool, these 
parietal cortical neurons exhibited both tactile and visual 
receptive fields (RFs) related to the animal’s hand: while the 
tactile RF was located on the hand skin, the visual RF was 
circumscribed to the visual space that closely surrounds the 
hand, the so-called peri-personal space. After the monkeys 
practiced and became proficient in the task of retrieving 
grapes with the rake, Iriki et al. observed that the visual RFs 
of the parietal neurons expanded to include the entire length 
of the rake, in addition to the peri-personal space around 
the animal’s hand. The Iriki laboratory interpreted these 
results as a suggestion that these cortical adaptations rep- 
resented modifications of the animal’s body schema that 
resulted in the incorporation of the rake as an extension of 
the animal’s arm, as seen from the brain’s own point of view 


(586). 


Long-term operation of BMIs that control the movements 
of artificial actuators, robotic or even virtual arms, leads to 
similar brain remapping of the receptive fields of cortical 
neurons located in multiple motor and somatosensory ar- 
eas, as described by Iriki in their experiments with tool 
usage. Several studies from our laboratory reported neuro- 
nal plasticity during learning to operate BMIs, starting with 
the study by Carmena et al. (114) that showed changes in 
neuronal tuning curves accompanied by changes in correla- 
tion between neurons as monkeys learned to operate a BMI 
that enacted reaching and grasping movements. Changes in 
neuronal tuning were further investigated by Lebedev et al. 
(463), and Zacksenhouse et al. (884) reported stronger cor- 
tical firing modulations during learning of BMI tasks, which 
decreased after monkeys learned. Transient increases in 
correlations between neurons, associated with learning a 
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bimanual BMI task, were confirmed by our laboratory 
(377). Overall, these studies showed that, after the mode of 
operation is switched to brain control, neuronal activity 
patterns markedly changed, both at the level of individual 
neurons and their populations. Changes in neuronal tuning 
were observed even when monkeys continued to perform 
arm movements during brain control. In this case, neuronal 
tuning to movements of their own arms weakened, and the 
neurons started to represent the BMI-controlled actuator 
instead (463). Moreover, neuronal tuning to the actuator 
remained even when monkeys stopped moving their own 
arms (114, 377, 463). At the population level, switching to 
brain control was associated with increased synchrony be- 
tween the neurons and, consequently, with many neurons 
having very similar preferred directions (114, 377, 583, 
598). 


Evaluation of changes in neuronal tuning during BMI op- 
erations has several caveats. The main factor that should be 
considered in such an analysis is that, during brain control, 
neuronal tuning properties no longer depend on the brain 
circuitry alone, but also essentially depend on the decoder 
settings. Indeed, the decoder uses a transfer function or an 
algorithm to translate the activity pattern of each neuron 
into a contribution to actuator movements. In the case a 
Wiener filter is used for decoding, the contribution of a 
neuron to a given degree of freedom is defined by the weight 
assigned to that neuron. For example, if a neuron is set to 
have rightward tuning, the decoder translates the discharge 
rate of that neuron into increments of the x-coordinate of 
the actuator, while no contribution is made to the y-coor- 
dinate. This decoder-assigned tuning may be different from 
the true neurophysiological properties of the neuron. Say, 
the neuron has switched to representing the y-coordinate 
and now matches the user’s intention to change the actua- 
tor’s y-coordinate and/or responds to the visual of the y-co- 
ordinate. Despite this new representation, the neuron will 
still contribute to the x-coordinate only because of the orig- 
inal decoder settings. In another scenario, the neuron fires at 
random and does not represent any intention or feedback, 
but still has a directionally tuned contribution established 
by the decoder. 


The interpretation of neuronal tuning during BMI control is 
further complicated by the ensemble properties of the neu- 
rons contributing to the decoding. Consider a Wiener filter 
(for simplicity with just one tap) applied to a population of 
randomly firing neurons. An assessment of directional 
properties of each neuron in the population would show 
cosine tuning with a preferred direction defined by the 
Weiner filter weights assigned to the x and y dimensions. In 
this example, neuronal tuning during BMI control does not 
necessarily match any neuronal representation of the user 
motor intention and/or sensory feedback from the actuator; 
the tuning only corresponds to the decoder settings. The 
next level of complexity to the analysis of tuning properties 


of a neuron during BMI control is brought by interferences 
from the other neurons. The BMI output is produced by 
many neurons, not just by the neuron whose tuning is being 
assessed. Therefore, when the firing of one neuron is com- 
pared with the BMI output, for example, cursor trajectory, 
this is effectively a comparison of activity of one neuron 
with a variable composed from the activity of many other 
neurons. Consequently, the tuning assessment critically de- 
pends on the relative contribution of different neurons. 
Two extreme cases can be considered: 1) the neuron in 
question has a very strong contribution to the decoder out- 
put whereas the contribution of the other neurons is rela- 
tively small, and 2) the neuron’s contribution is very small 
and the decoder output is dominated by the other neurons. 
In the first case, the neuron’s tuning will mostly represent 
the decoder settings for the reasons explained above. In the 
second case, the neuron’s tuning will reflect the relationship 
of that neuron’s firing to the actuator position generated by 
the other neurons, irrespective of the decoder weights as- 
signed to that neuron. Furthermore, correlated firing be- 
tween neurons may have strong effects on the BMI output 
and consequently on the tuning of individual neurons. For 
example, if activity of a weakly contributing neuron is cor- 
related with the activity of strongly contributing neurons, 
the tuning properties of the former will be very similar to 
those of the latter. Overall, although characterizing neuro- 
nal patterns during BMI control using tuning curves is help- 
ful to reveal some basic features (114, 317, 463, 598), in- 
terpretation of such tuning characteristics is not trivial. 


The pitfalls of neuronal tuning analysis for BMIs can be 
illustrated by the study of Ganguly and Carmena (283) that 
attempted to characterize the formation of new “cortical 
maps” as the result of learning to control a BMI. In that 
study, monkeys performed a two-dimensional center-out 
task using a BMI based on the recordings from small (~15 
neurons) M1 ensembles. The small-ensemble activity was 
translated into cursor position using a Wiener filter with 10 
taps. The study claimed that if the decoder is trained on day 
one and fixed afterwards, M1 neurons would plastically 
adapt to improve BMI performance and form a “cortical 
map.” The authors argued that the same M1 ensemble 
could simultaneously hold several “cortical maps” corre- 
sponding to different decoder. The “cortical map” was de- 
fined asa set of directional tuning curves, one per neuron. A 
close examination of this analysis reveals that neuronal tun- 
ing was determined differently from how it was set by the 
decoding algorithm. The 10-tap Wiener filter (100 ms bin 
width) effectively assigned 10 tuning curves for each neu- 
ron, one per tap. Yet, the authors chose to compute one 
tuning curve per neuron, which was derived either from a 
relatively short time window (200 ms) or a long one (2 s). 
The tuning curves were normalized to change from —1 to 1, 
which made it impossible to compare tuning strength in 
different neurons. Factors like the relationship of these tun- 
ing curves to the fixed decoder settings, relative contribu- 
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tion of different neurons to BMI output, and neuronal cor- 
relations were not considered. As training on the BMI task 
continued for 9-19 days, the tuning curves changed during 
the initial training days and later stabilized, which was in- 
terpreted as the formation of a “cortical map.” These 
changes were paralleled by a clear evolution of cursor 
movements, which started as highly convoluted trajectories 
resembling random walk, but changed to almost straight 
center-out trajectories during the late training days. This 
meant that the early and late tuning curves were generated 
from very different cursor movements, which by itself can 
explain the differences that appear as changes in neuronal 
tuning. For example, the 2-s time window most definitely 
contained movements in very different directions during the 
early days, and represented a more uniform sample during 
the late days, an analysis that is guaranteed to generate 
different looking tuning curves. Therefore, the seemingly 
paradoxical result that neuronal tuning curves changed for 
a fixed decoder that presumably would have kept them very 
stable, most likely reflected changes in cursor movement 
patterns rather than any meaningful characteristics of neu- 
ronal representation of the external actuator. Given these 
considerations, that study’s conclusion regarding the emer- 
gence of a “stable cortical map” appears questionable. A 
more plausible conclusion is that both BMI output and the 
underlying neuronal patterns changed during learning. 
More data would be needed to evaluate if there was any 
change in the cortical representation of the actuator move- 
ments resulting from learning to control the BMI. Specifi- 
cally, answers to the following questions would be needed: 
1) how the neuronal tuning curves are affected by the de- 
coder settings; 2) what other factors affect the tuning be- 
sides the decoder settings; and 3) how the tuning character- 
istics could be compared for datasets with very different 
actuator trajectories. 


In the above examples, the major difficulty in evaluating 
neuronal tuning during BMI control is related to a some- 
what circular approach: the actuator position is first gener- 
ated from neuronal activity using a mathematical algo- 
rithm, and then an attempt is made to determine the rela- 
tionship between the neuronal patterns and actuator 
movements once again, and to extract the features in this 
relationship that are not explainable merely by the decoder 
settings. This difficulty can be avoided if neuronal tuning is 
assessed based on parameter that is not generated by the 
decoder and can be manipulated independently of the de- 
coder settings. Such an analysis was conducted in our study 
of a BMI for bimanual movements (377). In that study, we 
evaluated neuronal tuning to target position, the parameter 
that unrelated to the decoder settings. Monkeys controlled 
2D movements of two virtual hands using a BMI; a separate 
target was designated for each hand. Since the target posi- 
tions were not included in the decoder variables, neuronal 
tuning to the targets could not be a consequence of the 
decoder settings. A k-nearest neighbor (k-NN) classifier 
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UKF decoder was used to extract the screen locations of 
targets from cortical ensemble activity on each behavioral 
trial, and the percentage of correct classifications was used 
as a measure of representation strength. The locations of the 
targets for both virtual hands were clearly represented by 
the cortical neuronal ensemble. These representations per- 
sisted during brain-control trials and passive observation 
trials. The passive observation trajectories did not change 
day to day. Therefore, we used them to assess long-term 
changes in the neuronal responses to the virtual hands. This 
analysis was valid because passive observation trials did not 
involve BMI control. The analysis was conducted offline by 
applying a UKF decoder to the neuronal recordings and 
using decoding accuracy as a measure of tuning strength. 
We found a clear improvement in decoding accuracy across 
the training days. 


Several studies evoked learning (and related plasticity) by 
altering BMI decoder settings and observing behavioral and 
neural adaptions to such manipulations. Thus Chase et al. 
(127) examined a BMI that generated 2D cursor position 
from monkey M1 activity using a linear decoder (127). 
Next, they applied a rotational transformation to the con- 
tribution to the BMI output from a subset of neurons. Al- 
though this manipulation initially resulted in curved cursor 
trajectories, their monkeys adapted to the new condition 
and straightened the trajectories. The analysis of neuronal 
responses showed that the entire neuronal population con- 
tributed to that adaptation, not only the neuronal subset 
with perturbed BMI outputs. Using a similar manipulation, 
Ganguly and Carmena (283) perturbed BMI output by ran- 
domly shuffling neuronal inputs to a fixed Wiener filter. 
Their monkeys successfully adapted to that perturbation. 


Sadtler et al. (684) devised a method that made adaptation 
to BMI control particularly difficult. They applied a factor 
analysis to extract correlated neuronal responses and rep- 
resent them as an intrinsic manifold, a subspace in a multi- 
dimensional space of population firing rates. The authors 
found that monkeys successfully learned to control the BMI 
with the inputs taken from the manifold, but learned with 
great difficulty if the inputs came from the outside of the 
manifold. In other words, monkeys adapted to a new de- 
coder if it did not require them to alter the original structure 
of neuronal correlations. Although this study seems to sug- 
gest an existence of strong synergies between the neurons in 
an ensemble, there is also an alternative explanation. The 
study utilized a threshold crossing method for detecting 
multiunit spikes, a method prone to inclusion of noise into 
the spike data. The noise most likely ended up outside the 
intrinsic manifold, so in the outside-of-manifold task mon- 
keys were asked to control the BMI with noise, obviously a 
task impossible to learn. 


In addition to brain plasticity induced by learning to oper- 
ate motor BMIs, plasticity occurs after training with sen- 
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sory BMIs (see the sections below on sensory and bidirec- 
tional BMIs). Studies conducted in our laboratory (341, 
798) enabled rats to perceive infrared light using a BMI that 
converted the signals from the head-mounted infrared sen- 
sors into ICMS of the rat primary somatosensory cortex 
(S1). Learning to use this BMI resulted in the emergence of 
a representation of infrared light in $1. Moreover, this new 
representation coexisted with $1 representation of the rat 
whiskers. 


Notwithstanding the progress in studies on BMI-induced 
neuronal plasticity, this research is still at the initial stage. 
Much more work will be needed to further clarify our un- 
derstanding of this phenomenon. Yet, if one would have to 
choose one consensual view in this field, it is the assumption 
that without the occurrence of some level of cortical plas- 
ticity, BMIs would not be able to operate as successfully as 
they do. In other words, after a decade and a half of intense 
work, BMIs certainly owe their prominence in systems neu- 
roscience to the exuberant propensity of the adult mamma- 
lian cortex to adapt itself to new task contingencies, partic- 
ularly when exposed to rich feedback signals. 


Vill. NONINVASIVE BMIs 


A. EEG-Based BMIs 


EEG-based systems are the most popular noninvasive 
BMIs, which have been thoroughly studied in humans, in 
both healthy subjects and patients. While approaches to 
EEG decoding are somewhat different from those used for 
extracting motor commands from streams of neuronal 
spikes, the general principles of neuronal ensemble physiol- 
ogy (583, 586) still apply to these applications. For exam- 
ple, decoding accuracy improves when more EEG channels 
are added (141). 


Although EEG signals are prone to be contaminated with 
many sources of noise, including facial EMG, electrooculo- 
gram (EOG), and all sorts of movement artifacts, and de- 
spite the fact that EEG recordings do not yield detailed 
motor information compared with intracranial single-unit 
recordings, EEG-based BMIs have been successfully imple- 
mented in both normal and disabled human subjects to 
enact motor commands and provide communication chan- 
nels. In particular, these EEG-based BMIs have been ex- 
tremely useful in allowing “locked in” patients, those suf- 
fering from a complete level of body paralysis, as a result of 
a neurodegenerative disorder such as amyotrophic lateral 
sclerosis (or Lou Gehrig’s disease), to regain the ability to 
communicate with the external world, using EEG based 
spelling devices (71-73, 129, 184, 358, 364, 455, 481, 573, 
606, 635, 718, 740, 742). 


The BMI classification into independent (endogenous) or 
dependent (exogenous) systems (see sect. III) is particularly 
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distinct for EEG-based systems. In independent BMIs, sub- 
jects perform volitional mental tasks, for example, motor 
imagery, that evoke changes in their EEG rhythms (3, 5, 17, 
27-29, 31, 51,77, 408, 560, 611, 632, 654, 811, 813, 826). 
The EEG bands typically employed in such BMIs are slow 
cortical potentials, mu (8-12 Hz), beta (18-30 Hz) and 
gamma (30-70 Hz) waves (14, 74, 815, 863). The majority 
of EEG-based BMIs translate EEG activity into discrete 
choices (17, 110, 159, 355, 370, 667, 669, 722, 840), but 
continuous control is also possible with independent EEG- 
based BMIs (211, 864, 893). 


Dependent EEG-based BMIs utilize computer screens or 
LED displays as sources of visual stimuli that evoke EEG 
responses. Users modulate these responses to produce a 
BMI output (10, 42, 70, 74, 101, 117, 521). Classification 
algorithms identify cortical responses to screen stimuli to 
which participants attend (overtly or covertly). For exam- 
ple, a decoder based on visual evoked potentials (VEPs) 
(162, 196, 216) exploits the fact that VEPs are stronger in 
the visual areas when subjects attend to the stimulus and/or 
look at it (147, 216, 555, 566). The P300 component of the 
response to a visual stimulus, also called P3 (206, 637), has 
been particularly popular in BMI designs (10, 15, 16, 34, 
54, 209, 243, 246, 257, 285, 363, 406, 521, 538, 545, 590, 
717, 735, 736, 787, 832) because of its high sensitivity to 
subjects’ reaction to the stimulus in “oddball” paradigms, 
where a person is required to detect a target within a train of 
irrelevant stimuli (12, 206-208, 778). Auditory-based (58, 
332, 410, 440, 455, 643) and tactile-based (95, 514, 814) 
P300 interfaces have been implemented, as well. Another 
popular BMI design is the one that utilizes VEPs generated 
by rapidly occurring stimuli (up to 60 Hz) (8, 42, 117, 122, 
186, 187, 386, 395, 473, 481, 502, 515, 868, 891). Such 
VEPs are called steady-state visual evoked potentials 
(SSVEPs). In a typical SSVEP-based BMI, multiple flicker- 
ing objects are shown on the screen; the flicker frequency is 
unique for each object. Subjects look at a specific object to 
issue a BMI command. 


Hybrid schemes for EEG-based BMIs have also been devel- 
oped (561, 626). Such BMIs combine several decoding prin- 
ciples, for example, motor imagery combined with SSVEPs 
(13, 100) or steady-state somatosensory evoked potentials 
(7), functional near-infrared spectroscopy (fNIRS) with 
asynchronous sensorimotor rhythms, P300 with SSVEP 
(154), and eye position with EEG decoding (337, 387, 833). 
Such hybrid BMIs are more versatile and accurate com- 
pared with BMIs that use only one control mode. 


Overall, EEG-based BMIs have successfully achieved many 
significant milestones. These include spelling devices (10, 
71, 101, 117, 132, 438, 559, 608, 619, 672, 876, 882), 
speech generators (190, 326, 549), BMIs for control of a 
humanoid robot (55, 103, 120, 138, 301), telepresence sys- 
tems (130, 223, 804), BMI-controlled wheelchairs (121, 
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169, 281, 411, 511, 663, 833), operation of a hand orthosis 
under BMI control (434, 436, 534, 610, 628), and BMI- 
controlled leg exoskeletons (156, 205, 282, 425, 456, 768, 
812). 


Notwithstanding these successes, it has been noted that 
many publications on EEG-based BCIs do not contain in- 
formation on how EEG artifacts were handled (244). This is 
a serious problem because artifacts not only contaminate 
EEG recordings, but they could serve as a source of control 
signal for a BCI. Curiously, users can easily utilize their 
facial EMGs, produced by clenching the teeth and recorded 
with regular EEG electrodes placed on the scalp, to control 
a robotic arm. In fact, this EMG control outperforms EEG 
control in the same settings (160). 


While EMG artifacts can be partially filtered out from EEG 
recordings by removing high-frequency signals (46, 311), 
mechanical artifacts strongly affect the EEG low-frequency 
range (331). Low-frequency mechanical artifacts can jeop- 
ardize the performance of BCIs that are based on slow cor- 
tical potentials. For example, José Contreras-Vidal and his 
colleagues employed linear regression decoders to recon- 
struct three-dimensional hand kinematics (86) and leg kine- 
matics during treadmill walking (652) from slow cortical 
potentials. No artifact removal was performed, which 
opens a possibility that mechanical artifacts could influence 
the reconstruction. Strong objections to these results were 
expressed by Castermans et al. (116) who showed that 
EEGs recorded during treadmill walking were contami- 
nated by the harmonics of the stepping frequency. In addi- 
tion to questioning the reconstruction of movements froma 
low-delta EEG band, this study found that mechanical ar- 
tifacts covered a wide range of EEG frequencies, so artifact 
removal by frequency filtering appeared to be unreliable. 
Yet another study (33) claimed that the employment of 
linear regression methods to reconstruct movements from 
cortical slow potentials was statistically invalid, since simi- 
lar reconstructions could be obtain from both real and ran- 
dom EEG data. 


B. Magnetoencephalography-Based BMIis 


In addition to electrical fields emitted by the brain, nonin- 
vasively recorded brain magnetic fields have also provided 
signals for BMIs (104, 413, 458, 541). Magnetoencepha- 
lography (MEG) detects weak magnetic fields produced by 
the electrical currents generated mainly by cortical neurons 
(149, 150, 334). Brain magnetic fields are very weak, on the 
order of picoTesla, or 10 million times less than the Earth’s 
magnetic field. To detect these tiny signals, one needs to 
employ very sensitive magnetometers. Historically, the cop- 
per induction coil was the first magnetometer (150), later 
replaced by superconducting quantum interference devices 
(SQUIDs) (149). Recently, atomic magnetometry was intro- 
duced based on a spin-exchange-relaxation-free (SERF) 
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magnetometer that is more sensitive than SQUID (871). 
MEG has better temporal and spatial resolution compared 
with EEG, but they can be used only in magnetically 
shielded facilities. 


Salmelin and Hari (688) reported suppression of the mu- 
rhythm, recorded with MEG, by thumb movements. Geor- 
gopoulos and his colleagues (296) employed a 248-sensor 
MEG to reconstruct arm movements from MEG recordings 
using a linear decoder. The first real-time MEG-based BMI 
was developed by Lal et al. (458); the system applied a 
binary classifier to MEG mu-rhythm. Mellinger et al. (541) 
used a similar design of a MEG-based BMI to classify mu 
and beta rhythms. MEG-based BMIs have been imple- 
mented in tetraplegic (413) and stroke (104) patients. 


C. fNIRS-Based BMis 


fNIRS applies light in the near-infrared range (600-1,000 
nm) through the skull to detect changes in oxyhemoglobin 
and deoxyhemoglobin concentrations in the brain blood 
flow (252, 388). fNIRS-based BMIs have been gaining pop- 
ularity recently (366, 533, 568, 744). fNIRS measures cor- 
tical metabolic activity with a spatial resolution of ~1 cm 
and temporal resolution on the order of 100 ms. However, 
the delay between neural activity and blood oxygenation 
changes is several seconds. 


The first fNIRS-based BMI was demonstrated by Coyle et 
al. in 2004 (166). That BMI extracted brain signals related 
to motor imagery when patients were asked to squeeze a 
ball. For this purpose, a single optode was placed over the 
motor cortex representation of the patient’s hand (EEG 
coordinate C3). Increases in oxyhemoglobin and decreases 
in deoxyhemoglobin were detected when subjects imagined 
the contralateral hand movements. The neurofeedback gen- 
erated by the patient’s motor imagery was provided by a 
variable-diameter circle shown on the screen. The BMI had 
75% accuracy in recognizing the patient’s imagined move- 
ments. Approximately 5 s were required to register the 
change in the hemodynamic response. 


Sitaram et al. (746) employed multiple optodes (four illu- 
minators and four detectors for each hemisphere) placed 
over the motor cortex. The subjects performed finger tap- 
ping with the left hand or right hand, or imagined these 
movements. Support vector machine (SVM) and Hidden 
Markov Model (HMM) algorithms allowed this BMI to 
achieve greater than 80% accuracy in decoding which fin- 
ger, right or left, was moving. 


A number of studies employed fNIRs to recognize prefron- 
tal cortex activity related to performing mental arithmetic 
(52, 53, 375, 649, 650). Additionally, fNIRs of prefrontal 
cortex activity was employed to decode subjective prefer- 
ence (517), music imagery (241, 649, 650), and emotional 
states (786). 
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As mentioned above, fNIRS can be used in combination 
with other neural recording methods to create hybrid BMIs. 
For example, hybrid EEG-fNIRS BMIs have improved the 
speed of fNIR thanks to the EEG’s superior temporal reso- 
lution. In addition, this hybrid method has allowed the 
generation of a larger repertoire of commands because dif- 
ferent types of brain-related signals are employed in the 
discrimination performed by this BMI. For example, Fazli 
et al. (248) recorded cortical sensorimotor rhythms simul- 
taneously with EEG and fNIRs methods, which resulted in 
better classification of motor imagery. Khan et al. (424) 
positioned the fNIRS sensors over the prefrontal cortex, 
whereas the EEG electrodes were placed over the motor 
cortex. This BMI processed 1) brain activity generated by 
mental arithmetic, which was detected from prefrontal re- 
cordings, and 2) motor commands generated by hand tap- 
ping, which were extracted from motor cortical activity. 


Interestingly, NIRS methods can detect not only slow re- 
sponses related to hemodynamics but also fast responses 
(with a millisecond temporal resolution) related to the scat- 
tering of light by neuronal membranes (315, 762, 861). This 
signal is weak, however, and has not been used for single- 
trial detection of neural activity. 


D. Functional MRI 


Functional MRI (fMRI) is another method for measuring 
brain hemodynamic responses (165, 276, 509, 772). fMRI- 
based BMIs derive their control signals from blood oxygen 
level dependent (BOLD) activity measured with an MRI 
scanner (744, 745, 847). fMRI has a relatively low temporal 
resolution (1-2 s), and the lag between neuronal activity 
and BOLD response is on the order of 3-6 s. The main 
advantage of {MRI-based systems is their spatial resolution 
that allows monitoring the entire activity of the brain. 
fMRI-based BMIs typically use slices with 5 mm thickness; 
each slice is represented by a 64 X 64 image with 3- to 
4-mm voxels. 


fMRI-based BMIs usually utilize visual feedback to display 
to subjects their own brain activity. For this purpose, vari- 
ous displays have been used, including functional brain 
maps (877), scrolling graphs of BOLD activity (139, 848), 
and virtual reality (745). The great advantage of {MRT is the 
ability to target a localized area as the source of a BMI 
signal. Weiskopf et al. (848) and Caria et al. (113) utilized 
imaging of the anterior cingulated cortex to develop a BMI 
for self-regulation of emotional processing. Additionally, 
BMIs have been developed for self-regulation of activity in 
the supplementary motor area (SMA) and Broca area (745). 


Several studies have employed {MRI-based BMIs to control 
computer cursors, avatars, and robotic arms. In the study of 
Yoo et al. (879), subjects were asked to perform several 
mental tasks (sequential number subtraction, covert speech, 
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and imagery of right or left hand clenching) to generate 
BOLD activity that drove a cursor through a two-dimen- 
sional maze. Yukiashu Kamitani and colleagues extracted 
fMRI representation of individual finger movements and 
drove a robotic hand with those signals, as reported in 
Scientific American (365). In Lee et al. (472), vertical and 
horizontal movements of a robotic arm were generated 
from fMRI signals. Cohen et al. (151) demonstrated control 
of a whole body human avatar in virtual reality by an fMRI- 
based BMI driven by motor imagery. Although control of 
prosthetic limbs from an fMRI scanner is not applicable to 
everyday use, such systems can be useful as rehabilitation 
tools. Additionally, {MRI-based BMIs can be potentially 
used as clinical devices for treating neurological conditions, 
such as stroke (185, 877), chronic pain (140), emotional 
disorders (113), and psychiatric disorders (430). 


IX. BMIis WITH ARTIFICIAL SENSATIONS 


A. Restoration of Sensations 


Sensory BMIs enable the flow of information from the ex- 
ternal world to be delivered back to the subject’s brain (57, 
199, 466, 469, 583, 678). These systems strive to repair 
damage to sensory neural circuitry. In principle, sensory 
BMIs could interfere with different levels of neural sensory 
processing, from peripheral receptors to the spinal cord, 
brain stem nuclei, thalamus, cortical sensory areas, and cer- 
ebellum. 


For the development of efficient sensory BMIs, it is impor- 
tant to understand that sensory processing does not depend 
only on a unidirectional flow of information from the pe- 
ripheral receptors or sensory organs to hierarchically higher 
processing stages. Top-down modulatory signals (e.g., de- 
scribing influences to the lower-order areas from the higher- 
order brain regions) are vital for sensory processing in 
awake subjects (303, 306, 454, 464, 617, 739). Such mod- 
ulatory signals are essential during the execution of volun- 
tary movements (126, 572, 714-716, 756, 760) and active 
sensory exploration of the environment (176, 314, 454, 
615, 617). 


Sensory impairments can take many forms, from a complete 
loss of sensation caused by destruction of peripheral recep- 
tors and nerves to impairments of certain aspects of sensory 
processing that occur when cortical or subcortical areas are 
damaged. For example, following extensive lesions to the 
primary visual cortex, patients (163, 769, 849) and mon- 
keys (164) do not perceive visual stimuli, but may retain an 
ability to utilize visual information. This phenomenon, 
called blindsight, is mediated by subcortical visual struc- 
tures like the superior colliculus. Additionally, damage to 
cortical areas of the so-called ventral visual stream produces 
deficits of visual object recognition, whereas damage to the 
dorsal stream areas impairs spatial visual processing and 
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visually guided movements (312, 313, 346). While such 
peculiar sensory disabilities could likely be treated with 
BMIs in the future, current implementations of sensory 
BMIs deal mostly with cases of damage to peripheral sen- 
sory receptors, sensory nerves, or spinal tracts. In these 
cases, there is a loss of normal sensation, but higher-order 
sensory areas remain intact and could still process sensory 
information if it is delivered to them using a BMI. Accord- 
ingly, sensory BMIs attempt to mitigate the devastating ef- 
fects of peripheral lesions by linking these intact brain areas 
to artificial sensors. 


We will focus on sensory BMIs that enable artificial tactile 
sensations. In such BMIs, electrical stimulation is usually 
used to reactivate sensory responses (260, 598, 676, 784). 
Additionally, several papers have recently employed opto- 
genetic methods to induce somatosensory sensations (536, 
875). Stimulation can be applied to somatosensory cortex 
(368, 598, 676, 784), thalamus (182, 321, 479, 605), and 
peripheral sensory nerves (183, 662, 683, 791). 


Tactile sensations evoked by electrical stimulation of the 
surface of the postcentral cortex without eliciting move- 
ments were first described in 1909 by Harvey Cushing 
(178); they were later extensively studied by Wilder Penfield 
(623). Penfield’s patients most often reported sensations of 
numbness or tingling, rarely pain. The modern era in this 
research started with the experiments of Ranulfo Romo et 
al. (676) who employed small currents injected through a 
microelectrode, the method called ICMS, to evoke tactile 
sensations in monkeys. Romo’s monkeys started with learn- 
ing a sensory discrimination task where they compared two 
vibrotactile stimuli applied to their hands one after another. 
The animal reported, by pressing a button with an opposite 
hand, which of the two vibrations had a higher frequency. 
Next, the first stimulus in the sequence remained vibrotac- 
tile, whereas the second one was an ICMS train applied to 
$1. The task was again to compare the frequencies at which 
the stimuli were presented. Surprisingly, monkeys began to 
successfully compare the vibrotactile and ICMS patterns 
with very little practice. This result suggested that sensa- 
tions resembling skin vibration could be evoked artificially 
with ICMS of S1. Romo et al. (676) penetrated $1, with a 
microelectrode placed at a new location every day; they did 
not implant those microelectrodes. With this method, they 
could not study long-term changes in the ICMS-induced 
artificial sensations. 


A long-term study of ICMS effects with implanted micro- 
electrodes was conducted by our laboratory (260) (FIGURE 
17). The experiments were conducted in owl monkeys 
chronically implanted with cortical microelectrode arrays. 
The experimental task consisted of having animals reach 
and open one of two doors. Animals were searching for a 
piece of food that was hidden behind one of the doors. In 
each trial, the location of the food was cued by an ICMS 
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train. Progressively, more complex ICMS patterns were em- 
ployed as the animals learned novel tasks. Monkeys were 
first required to simply detect the presence of ICMS. Next, 
they had to discriminate temporal patterns of ICMS, and 
finally they discriminated spatiotemporal ICMS patterns 
delivered through multiple electrodes. Although it took sev- 
eral weeks for monkeys to learn the initial, simple task, their 
ability to interpret new ICMS patterns clearly improved 
after several months of training with ICMS. After this initial 
learning phase, animals could acquire a new and more dif- 
ficult task in just a few days. This result indicated that ICMS 
progressively generated a new sense, some sort of artificial 
touch sensation, that monkeys could readily utilize. In fact, 
it seems that some degree of generalization was achieved by 
these monkeys after a few months of training, which al- 
lowed them to learn new tasks that involved ICMS faster 
than when they were naive in terms of experiencing ICMS. 
We also conducted a study in rhesus monkeys where ICMS 
of the $1 instructed the animals about the direction of joy- 
stick movement they had to produce in a trial (597). Like 
the owl monkey experiment, rhesus monkeys learned that 
task after several days of training. 


Talbot et al. (784) asked if sensations from different skin 
locations could be evoked using ICMS. They applied ICMS 
through different microelectrodes of the arrays implanted 
in the hand representation of $1. Monkeys reported with 
eye movements where on the hand they felt the stimulus. 
Predictably, these experiments confirmed the well-known 
S1 somatotopic organization (394). Additionally, it was 
determined that monkeys could discriminate ICMS inten- 
sity and match it to the pressure applied to the hand using a 
mechanical probe. Virtually the same experiment was re- 
cently conducted in a tetraplegic patient implanted with a 
Utah array in S1 (262). The patient correctly matched cor- 
tical stimulation sites to different hand locations. Stimula- 
tion applied through ECoG grids has been shown to evoke 
somatosensory sensations, as well (589, 638). This stimu- 
lation method evoked sensations of tingling, numbness, and 
temperature. Electrical stimulation of both the precentral 
and postcentral locations was effective in producing these 
sensory effects. 


Several studies explored peripheral nerve stimulation as a 
method to provide humans with artificial tactile sensations. 
One study (791) employed peripheral nerve cuff electrodes 
implanted in two patients with arm amputation for more 
than 1 yr. Patterned electrical stimulation of the nerves 
produced touch perceptions in the phantom hands that the 
patients described as being natural (tapping, pressure, mov- 
ing touch, and vibration); the sensations changed with mod- 
ifications in the stimulation pattern. These artificial sensa- 
tions improved the subjects’ performance with a prosthetic 
hand. In the other study (183), phantom hand sensations 
were evoked in amputees using electrical stimulation of the 
median or ulnar nerve delivered through a 96-channel Utah 
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arrays, which remained implanted for 30 days. The same 
stimulation approach was utilized to reproduce sensations 
froma hand prosthesis that performed grasping tasks (662). 


Several somatosensory BMIs have been demonstrated in 
rats, taking advantage of the exquisite tactile skills of these 
animals. A study of John Chapin’s laboratory (790) re- 
ported a BMI that guided rat navigation through three- 
dimensional structures. Steering cues were provided by 
ICMS of $1, whereas locomotion was reinforced by ICMS 
applied to the medial forebrain bundle, a structure known 
to be part of the brain’s reward system. A human operator 
used this BMI to steer rats over complex terrains. Venka- 
traman and Carmena (820) developed an active sensing 
paradigm based on ICMS of rat $1. ICMS was delivered 
when a whisker crossed a spatial location designated as a 
target. Rats were rewarded for localizing the invisible target 
and crossing it several times with the whisker. 


Using rats as an experimental model, Thompson et al. (798) 
showed that ICMS of S1 could be used to substitute or 
augment the animal’s natural vision. Their BMI allowed 
rats to use the S1 cortex to perceive, or “touch,” otherwise 
invisible infrared light. Light from infrared (IR) sources was 
detected by head-mounted sensors and converted into 
ICMS applied to the rats’ S1 representation of their facial 
whiskers. Initially, rats took 4 wk to learn to use a BMI with 
a single infrared detector to find reward ports that emitted 
infrared light. An upgrade to this system included four IR 
sensors that provided a panoramic infrared vision (341). 
Using this system, a new group of rats took only 3 days on 
average to find the same infrared sources. After rats learned 
to utilize this BMI, electrophysiological recordings revealed 
that S1 neurons developed multimodal receptive fields that 
represented both somatosensory responses from the facial 
whiskers and infrared light generated in the animal’s sur- 
roundings. These results showed that, even in adult ani- 
mals, primary cortical areas can incorporate new sensory 
representations, leading to the emergence of multiple and 
overlapping sensory maps simultaneously sustained by the 
same neuronal populations. 


Sensory substitution through haptic stimulation of the sub- 
ject’s body is an alternative to using the neurostimulation 
approach, which is particularly relevant to neurorehabilita- 
tion. A study by the Walk Again consortium (737) used 
haptic stimulation to restore the sensation of autonomous 
walking to paraplegic patients. For this purpose, a new 


paradigm was developed for reproducing lower limb so- 
matosensory feedback in paraplegics by substituting sensa- 
tions generated by a haptic display placed on patients’ fore- 
arms for the normal sensation generated by walking legs. 
Initially, leg movements were simulated by making an ava- 
tar of the patients move in an immersive virtual reality 
environment. Patients used goggles to observe their avatars 
moving on different ground surfaces while a haptic display 
was used to deliver a wave of tactile stimulation to the skin 
of their forearms. The use of this haptic display induced 
patients to experience the perception of walking on various 
surfaces: grass, a paved street, or beach sand. Moreover, 
patients perceived leg movements during the swing phase of 
the avatar legs and experienced the perception of their feet 
rolling on the floor, despite the fact that their legs were 
completely paralyzed. These results showed that virtual re- 
ality training combined with haptic stimulation resulted in 
the assimilation of the virtual lower limbs in the body rep- 
resentation present in the patients’ brains. These findings 
suggest that, in the future, the addition of rich haptic feed- 
back to rehabilitation devices will be essential to restore 
realistic perceptual experience in paralyzed patients. 


In addition to electrical stimulation, optogenetic stimula- 
tion has been steadily gaining popularity (875), so it is plau- 
sible that this method will be used in sensory BMIs in the 
future. Another stimulation method employs ultrasound 
(475, 674). Recently, implantable microcoils have been de- 
veloped for magnetic stimulation (474). 


B. Brain-Machine-Brain Interface 


Brain-machine-brain interfaces (BMBIs), also called bidi- 
rectional BMIs, perform both the extraction of motor com- 
mand signals from raw brain activity and the delivery of 
sensory feedback to the brain (57, 255, 469, 583) or periph- 
eral nerves (543). This approach was pioneered by our lab- 
oratory (597, 598) (FIGURE 13). In our experiments, rhesus 
monkeys were chronically implanted with microelectrode 
arrays in M1 and S1. M1 implants were used for the extrac- 
tion of motor commands, and ICMS was delivered through 
$1 implants. The motor loop of this BMBI controlled move- 
ments of an avatar arm shown on a computer screen placed 
in front of the animals. Monkeys used this avatar arm to 
actively explore a set of virtual objects (2 or 3 gray circles) 
rendered in the virtual space they searched. The objects 
were visually identical but differed in terms of their artificial 


FIGURE 17. 


Long-term experiments in owl monkeys on reaching movements cued by intracortical microstimulation of somatosensory cortex 


(S1). A: diagram of the experimental task. After a barrier was lifted, monkeys reached toward one of two doors; a food pellet was behind one 
of them. B: location of cortical implants. S1 implant was used to deliver microstimulation. C: microstimulation parameters. D-F: stimulation 
patterns. The first task (E) required reaching to the right if a sequence of microstimulation pulses was delivered. If no stimulation was applied, 
monkeys reached to the left. In the second task (G), the rule was reversed: monkeys reached leftward in response to microstimulation. The third 
task (D) employed two different temporal patterns of microstimulation. The fourth task (F) used spatiotemporal patterns of microstimulation 
produced using four pairs of implanted microwires. [Adapted from O’Doherty et al. (597).] 
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texture. Monkeys had to assess the objects’ texture by using 
the BMBI to scan their avatar hands over the surface of the 
virtual objects. When a monkey’s avatar hand came into 
virtual contact with the surface of a given object, a pattern 
of ICMS was delivered to the hand representation of the 
animal’s $1. Monkeys had to identify a specific virtual tex- 
ture using this BMBI and then hold the avatar hand over it 
to obtain a fruit juice reward. The implementation of this 
BMBI had a caveat: since ICMS evoked electrical artifacts 
that occluded the neuronal spikes, recordings and stimula- 
tion could not be conducted simultaneously. This issue was 
solved by switching from recording to stimulating every 50 
ms. Although this approach resulted in a loss of some neu- 
ronal data, the BMBI still performed well because several 
hundred neurons were recorded simultaneously. A similar 
BMBI was reported by Richard Andersen’s group (437). In 
that system, ICMS was applied to $1, whereas BMI control 
commands were extracted from PPC. 


Several recent studies have implemented bidirectional inter- 
faces with peripheral nerves. Davis et al. (183) demon- 
strated real-time control of a robotic finger by amputees 
using multielectrode recordings from the median or ulnar 
nerves. The decoding was performed by a Kalman filter. 
The same electrodes were used to deliver sensory feedback 
using electrical stimulation. The other study (662) reported 
a myoelectric interface that amputees could control using 
surface EMGs to produce grasping movements using a ro- 
botic hand. Grasp force feedback, produced by robotic sen- 
sors, was delivered using intrafascicular stimulation of the 
median and ulnar nerves; stimulation intensity was propor- 
tional to the sensor signal. This bidirectional setup enabled 
the subjects to maintain three force levels without looking 
at the robotic hand. 


In addition to BMBIs that provide sensory feedback from an 
external actuator, several recent demonstrations of closed- 
loop activity-dependent stimulation can be described as 
BMBIs. In these systems, neuronal activity is recorded from 
a brain area and then converted into a pattern of electrical 
stimulation delivered to the same or a different area. Such 
feedback loops may serve different purposes. Andrew Jack- 
son and his colleagues at Eberhard Fetz’s laboratory em- 
ployed a neural implant to form and strengthen an artificial 
connection between two sites in the motor cortex of freely 
behaving monkeys (381, 382). The implant triggered elec- 
trical stimulation in one cortical location with neuronal 
discharges recorded from a different site. Several days of 
operation of this implant produced a stable cortical reorga- 
nization that was evident from the changes in wrist move- 
ments evoked by electrical stimulation applied to each site. 
Wrist movements evoked from the implant’s recording site 
started to resemble those evoked from the stimulation site, 
which indicated that a Hebbian potentiation of synaptic 
connections occurred for the artificial connection. The au- 


thors suggested that this approach could be used for neu- 
rorehabilitation in the future. 


Lucas and Fetz (513) employed EMG-triggered cortical 
stimulation to induce a similar targeted reorganization of 
cortical motor output. They observed that the stimulated 
cortical site became associated with the activity of the re- 
corded muscle, even though that particular muscle was not 
represented by neurons in that cortical location previously 
(513). Yet another study by the Fetz laboratory (592) dem- 
onstrated that spinal stimulation, triggered from cortical 
spikes, could modify the strength of corticospinal connec- 
tions in a manner consistent with spike-timing-dependent 
plasticity. 


Several studies have shown that closed-loop stimulation 
systems can lead to partial recovery of function in neuro- 
logical conditions resulting from injury or disease. Guggen- 
mos et al. (330) employed a functional bridge connecting 
motor and somatosensory areas of the rodent brain to pro- 
mote recovery of motor skills after traumatic brain injury. 
McPherson et al. (540) used EMG-triggered spinal stimula- 
tion to facilitate recovery after spinal cord injury in rats. 
Overall, these studies showed that BMBIs could be used to 
plastically modify neural connectivity and promote func- 
tional recovery. Our laboratory developed a closed-loop 
stimulation system for epilepsy control (618). In that study, 
rats were treated with pentylenetetrazole to provoke epilep- 
tic seizures. The system detected the seizure episodes in 
cortical LFPs, and applied electrical stimulation to rat spi- 
nal cord to suppress the seizures. This approach reduced the 
frequency of seizure episodes by 44%. In the future, a sim- 
ilar approach may prove useful for the treatment for drug- 
resistant epilepsy. 


A stimulation system has been suggested as a potential pros- 
thetic system for improving memory (62, 63). In these stud- 
ies, a multiple-input, multiple-output model reproduced the 
associations between CA3 and CA1 regions of the rat hip- 
pocampus. Neuronal ensemble recordings were conducted 
in CA3 and CA1 of rats performing a delayed-nonmatch- 
to-sample memory task. A nonlinear MIMO was trained to 
predict CA1 activity based on CA3 patterns. The predicted 
patterns of activity were then delivered to CA1, using elec- 
trical stimulation through the same electrodes that recorded 
neuronal spikes. The stimulation improved task perfor- 
mance in normal rats and restored performance in rats with 
a pharmacological block of hippocampal synaptic trans- 
mission. The authors suggested that this approach could be 
used to restore long-term memory function in patients with 
damage to hippocampus and its interconnected structures. 


X. COGNITIVE BMis 


Cognitive BMIs or cognitive neural prostheses deal with 
brain activity related to higher-order functions, as opposed 
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to more simple motor and sensory functions (625, 792). 
Although the distinction between higher-order and lower- 
order functions is not clear cut, by convention BMIs are 
called cognitive if they work in the domains of cognitive 
states (348, 886), executive functions (349), decision mak- 
ing (21, 335, 343), memory (62, 63), attention (35, 417), 
and language (98, 99, 327, 482, 834, 839). 


Several intracranial cognitive BMIs have been developed, 
mostly dealing with decoding of different aspects of motor 
decisions during the periods of immobility preceding move- 
ment onsets. For example, Hasegawa et al. (343) decoded 
go versus no-go decisions, and the prepared saccade direc- 
tion from the activity of monkey superior colliculus neurons 
(343). Musallam et al. (564) decoded the representation of 
expected rewards and motor decision from the neural ac- 
tivity recorded in the cortical parietal reach region. In that 
study monkeys were engaged in an instructed-delay task 
where they prepared an arm movement, but withheld it for 
several seconds. In the beginning of each behavioral trial, 
the monkey was shown a cue that indicated what kind of 
reward would be given. A Bayesian algorithm was applied 
to decode expected reward and target location simultane- 
ously. 


In our laboratory, a BMI approach was employed to extract 
decisions involved in reprogramming a motor goal (376). 
Monkeys performed a center-out task, where they moved a 
cursor towards screen targets using a joystick. Neuronal 
ensemble activity was recorded from M1 and S1 arm rep- 
resentations. Monkeys started the trials by placing the cur- 
sor at the screen center. Next, a target appeared at an off- 
center location. In some trials, that initial target appeared 
for 50-250 ms, and then it was replaced by a new target at 
a different screen location. We found that both the emer- 
gence of the decision to move to the initial target and the 
new decision to cancel that motor plan and move to a new 
target could be decoded from population M1 activity. Tar- 
get locations were decoded using an LDA classifier. This 
analysis showed that M1 activity initially represented the 
first target, then simultaneously represented both targets, 
and eventually shifted to represent the new target only. 
Based on these findings, we proposed that such decoding of 
covert motor planning could improve motor BMIs by 
equipping them with the capacity to detect motor decisions 
early and inhibiting them if the user decides to cancel a 
prepared action or chooses a different one. 


In the other study, we decoded representation of time from 
M1 and PMd activity in the absence of over behavior (468). 
Monkeys were trained to perform self-timed button presses, 
where they touched a button with their hands, maintained 
contact for 3—4 s, and then released the button. We found that, 
while monkeys self-timed the required interval and did not 
produce any movements, their M1 neurons exhibited ramp- 
ing activity patterns. We then employed a Wiener filter to 
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decode the representation of temporal intervals from these 
M1 patterns. Such decoding of action timing could be useful 
for developing BMIs that enact typical motor behaviors 
where movements are intermingled with periods of immo- 
bility. 


Motor imagery, widely used in noninvasive BMI (5, 28, 
272, 551, 773, 840), can be considered a cognitive compo- 
nent of a BMI. Richard Anderson’s group recently decoded 
motor-imagery from the intracranial PPC recordings in a 
tetraplegic human (3). PPC is engaged in higher-order as- 
pects of motor behaviors (18). In the tetraplegic subject, 
motor imagery clearly activated different PPC neurons, de- 
pending on the specific action being imagined (e.g., imagin- 
ing hand movement to the mouth or ear, imagining shoul- 
der rotation, etc.). Additionally, PPC neurons responded to 
the imagery of movement goal, movement trajectory, and 
the type of movement. All these variables were successfully 
decoded from the activity of PPC neuronal populations. 
Moreover, the subject learned to control a robotic arm by 
imagining movements. 


As intracranial recording methods become more routine 
in clinical studies, we will probably see a rapid development 
in BMIs related to human cognitive processes. Advances in 
this new domain will likely contribute to the emergence of 
new clinical applications for BMIs, as well as the incorpo- 
ration of fundamental knowledge about the neurophysio- 
logical involved in higher brain functions. 


XI. BRAIN-TO-BRAIN INTERFACES AND 
BRAINETS 


The growth of BMI research gave rise to a large variety of 
spin off experimental paradigms. In one of the variations of 
the classical BMI approach (FIGURE 18), multiple animal 
(or human) brains can be connected to each other to estab- 
lish a direct brain-to-brain communication linkage, called 
brain-to-brain interface (BTBI) (616). In the other varia- 
tion, several individual brains collaborate on a common 
motor task by establishing a network of brains, or a Brainet 
(614, 657) (FIGURE 19). 


By definition, BTBIs allow multiple animals to exchange 
information using a protocol that incorporates both neural 
recording and stimulation. The pioneering BTBI was imple- 
mented in rats (616). In that study, the first animal per- 
formed the role of information encoder, and the second 
animal was the decoder of a simple binary message. The 
binary message represented the encoder rat performing a 
two-choice behavioral task (active tactile discrimination 
or responses to a visual stimulus). The encoder rat’s neu- 
ronal firing rates, recorded from either the S1 or M1, 
depending on whether the rat performed a tactile dis- 
crimination or a visuomotor task, underwent a sigmoid 
transform and then were converted into patterns of ICMS 


811 


is) 
(o} 
= 
> 
(eo) 
fed} 
[ok 
oO 
a 
=> 
(eo) 
3 
=F 
= 
go 
od 
go) 
=] 
< 
n 
aa 
oO 
= 
To 
= 
< 
w. 
a 
[e} 
© 
> 
(eo) 
= 
© 
as 
a 
< 
= 
2: 
ine) 
ye) 
= 
wo 
oo 
iN 
[o} 
= 
= 
fet) 
< 
vcd 
pe) 
oO 
= 
N 


LEBEDEV AND NICOLELIS 


M1 neural ensemble 


egdee SSNS 6) 1 MN SSS!) Gewese eee [) 
es) . toget tie ten Pied 4s: a mre oO 
eth © 0/0H) | RRP eR eee) Be Shtatet weiss QD 
del ih ee) wa) ele ' a u ' ' — 
Wi eeene ree 1 ao 
Te erent eee nT - 
' a 
8G ROE ew ee Oh) ORES ENED 18 Pete eee eee a ree) i= 
Sete wees ' = a ' ' meet =] 
‘ ' ' Zz 
eu wees mae 92 mn sau 
Correct 
Lever 
Feedback 


i 


Encoder 


(2nd reward) 


Sigmoid Transform 


ICMS 


fro aa 


> ea 


Correct 
Lever 


ZScore 


= 


Decoder 


FIGURE 18. Brain to brain interface. Two rats participated in the experiment, the encoder rat and decoder 
rat. The flow of information between the animals is shown by arrows. The encoder rat responded to a visual 
stimulus provided by an LED by pressing one of two levers. Activity of an M1 neuronal population activity was 
recorded while the encoder performed the task. This activity underwent a sigmoid transform to generate a 
microstimulation pattern, which was delivered to the somatosensory cortex of the decoder rat. The decoder 
animal had to select the same lever. The encoder rat received an additional reward if the decoder rat 


performed correctly. [From Pais-Vieira et al. (616). 


applied to $1 or M1 of the decoder rat. This latter animal 
could be located next to or far apart from the encoder. 
On average, the decoder reproduced the behavioral 
choices of the encoder rat in about 70% of the trials. 
During operation of the BTBI, the encoder rat received an 
additional reinforcement. Pais-Vieira and colleagues no- 
ticed that following an error by the decoder rat, the en- 
coder rat adapted both its behavior and cortical activity 
to generate cleaner neuronal signals to be broadcast to its 
partner. Invariably, the decoder rat performed better af- 
ter this encoder’s adaption. 


In the next study by Pais-Vieira et al. (614), several rat 
brains were connected to a network of brains - named a 
Brainet - that performed several elementary computations, 
like discrimination of ICMS patterns by several rats simul- 
taneously to improve overall discrimination accuracy, or 
retaining information in their collective memory by trans- 
ferring it from rat to rat (614). ICMS served as an input to 
such a Brainet, while the output was derived from cortical 
activity of the participating animals. Essentially, the Brainet 
acted as an organic computer that processed input data 
through a network of living brains. 


These initial publications were followed by a number of 
studies by different groups unified by a common theme of 
connecting of the brains of different organisms. Yoo et al. 
(878) connected the brain of a human to the spinal cord of 
a rat. The human operated an SSVEP-based BMI to gener- 
ate “go” commands to an anesthetized rat. The command 
was executed by applying transcranial focused ultrasound 
to the rat motor cortex, causing the movement of the ani- 


] 


mal’s tail (878). In another study BTBI connected two dif- 
ferent species (488). The human attempted to make the 
cockroach walk along an S-shape track, and succeeded in 
20% of cases. In the other study (727), the premotor cortex 
of one monkey was linked to the spinal cord of a second, 
anesthetized primate. The second monkey’s hand was at- 
tached to a joystick. The first monkey generated motor 
intention commands while looking at a computer screen 
that showed a cursor and a target of movement. This inten- 
tion command was extracted from premotor cortex activity 
and translated into a stimulation pattern applied to the 
spinal cord of the second monkey, causing the joystick 
movement, which in turn moved the cursor on the first 
monkey’s screen. A proof of concept study (265) showed 
that gene expression can be controlled by brain signals. In 
that study, a human operating an EEG-based BMI optoge- 
netically controlled the expression of designer cells. The 
designer cells were either in culture or in subcutaneous im- 
plants in mice. 


Several BTBIs have been demonstrated in humans. Grau 
et al. (316) had one human subject, the emitter, operate a 
motor-imagery EEG-based BMI. The binary output of 
that BMI was delivered to the brain of the second subject, 
the receiver, using TMS pulses applied to the visual cor- 
tex. Depending on whether the transmitted signal was 
“1” or “0,” a robot placed the TMS coil over the area 
where stimulation induced or did not induce a conscious 
perception of phosphenes. Information transfer rates of 3 
and 2 bits per minute were achieved for the BMI perfor- 
mance and message transmission, respectively. Rao et al. 
(661) employed a very similar BTBI design, with the dif- 
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FIGURE 19. Monkey brainet. A: diagram of the experimental setup. Up to three monkeys were seated in 
monkey chairs in separate rooms. Each monkey faced a computer screen that displayed a virtual avatar arm. 
The behavioral task consisted of reaching screen targets with the avatar arm. The avatar arm was controlled 
jointly by several monkeys. B: shared control task, where each of two participating monkeys contributed 50% 
to the (X, Y) position of the virtual arm. Locations of microelectrode arrays are shown below the task diagram. 
C: partitioned control task, where one monkey contributed to X position of the avatar arm, whereas the other 
monkey contributed to Y position. D: a 3-monkey task. Each monkey performed a two-dimensional task, and all 
three together controlled 3-dimensional movements of the avatar arm. [From Ramakrishnan et al. (657).] 


ference that TMS was applied to the motor cortex of the 
second subject. Accordingly, the second subject re- 
sponded with a TMS-induced hand movement that pro- 
duced a touchpad press. 


While the studies reviewed above emphasized direct com- 
munication between different brains, our laboratory re- 
cently demonstrated several Brainets that emphasized coop- 
eration of multiple subjects to achieve a common motor 
goal of a typical upper limb BMI (657). In that study, two or 
three monkeys shared control of the movements of an ava- 
tar arm using their combined cortical activity. Three Brainet 
designs were tested. The first design, called shared-control 
Brainet, merged the outputs of two monkey brains. Cortical 
activity of each monkey was processed by a separate de- 
coder. The decoder outputs were then averaged to set the 
coordinates of the avatar arm. Performance improvement 
was achieved because the averaging of contributions from 
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both monkeys enhanced the signal and suppressed the 
noise. In the second design, called partitioned control 
Brainet, two monkeys performed together again, but they 
had different tasks. The first monkey generated neural 
control commands to move the avatar arm in the hori- 
zontal dimension, while the other monkey controlled the 
vertical dimension. In that Brainet, performance im- 
proved because each monkey made fewer errors in the 
simple, one-dimensional task. In the third Brainet design, 
named a triad Brainet, three monkeys cooperatively con- 
trolled three-dimensional movements of an avatar arm. 
Yet, each monkey performed a two-dimensional task, 
and all animals were unaware that the cooperative task 
was three-dimensional in nature. That design modeled a 
“super-brain” that, by combining the brain activity of 
three individual brains into a single computing system, 
handles a higher-order task while individual brains have 
lower-order contributions. 
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Somewhat similar cooperative systems have been devel- 
oped using EEG-based controls by several humans. These 
include a BMI for spacecraft navigation controlled by 
two users (644), BMIs for group decision making (217, 
645, 883), and a cooperative BMI for movement plan- 
ning (841). 


Xil. BMI AS A POTENTIAL 
NEUROREHABILITATION THERAPY 


Since the late 1990s, when BMI research began in earnest, 
the field has focused primarily on achieving two major 
goals: 1) to establish a new paradigm to investigate the 
dynamic physiological properties of distributed neural cir- 
cuits in behaving animals, and 2) to explore the possibility 
of creating new assisted technologies, aimed at restoring 
upper, lower, or full body mobility in severely paralyzed 
patients. For the past decade, the focus on developing clin- 
ical applications based on BMIs has increased markedly, as 
noted throughout this review. Yet, no one had anticipated 
that this paradigm could provide benefits beyond the com- 
monly stated goal of assisting patients in regaining mobility 
through the employment of a new generation of brain-con- 
trolled prosthetic or orthotic devices. Thanks to recent clin- 
ical studies, however, a third potential future application of 
this paradigm has been introduced: the use of BMIs as a 
neurorehabilitation tool (26, 27, 82, 204, 205, 737, 741, 
751, 752, 819). 


To date, noninvasive BMIs have been used as neuroreha- 
bilitation tools primarily in clinical studies focused on 
stroke victims. The main assumption motivating these stud- 
ies has been that practice with a BMI that mimics move- 
ments of a paralyzed limb could facilitate brain plasticity 


A B 


Mean improvement 
after 10 months of training 


and contribute to some level of motor recovery. For exam- 
ple, stroke patients can learn to operate an MEG-based BMI 
by modulating their « rhythm recorded in the hemisphere 
ipsilateral to the lesion (104). In this study, the BMI opened 
and closed an orthosis that was attached to the paralyzed 
hand. This learning did not cause noticeable clinical im- 
provements. However, long-term BMI training combined 
with physical therapy resulted in clear motor recovery (93, 
659). As shown by the analysis of motor evoked potentials 
(MEPs), the recovery was related to enhanced neuronal ac- 
tivity in the hemisphere ipsilateral to the stroke site (87). 
Similar results were demonstrated by a study that combined 
a BMI-controlled robot with robot-assisted physical ther- 
apy (27, 30). Combining BMI training with virtual reality 
resulted in clinical improvements as well (64). Additionally, 
a combination of BMI control with transcranial direct cur- 
rent stimulation (tDCS) showed positive clinical results 
(753). 


Much less research has been conducted on the effectiveness 
of BMI training in patients with SCI. In the first long-term 
study of this kind, Donati et al. (205) conducted BMI train- 
ing of eight chronic paraplegic patients in a multi-stage 
rehabilitation paradigm, aimed at restoring bipedal loco- 
motion through robotic lower limb orthoses. The core of 
this paradigm was based on the utilization of an EEG-based 
BMI that allowed patients to control multiple actuators, 
ranging from avatar bodies to two types of robotic walkers: 
a commercially available gait robotic system (Lokomat) 
(384) and a custom-designed lower limb exoskeleton. In 
addition to the traditional visual feedback, this BMI was 
also coupled with a haptic display system that delivered 
continuous streams of tactile information to the skin of the 
patients’ forearm. These artificial tactile/proprioceptive sig- 


Example of expansion of the zone of 
partial preservation (ZPP) for sensory 


assessment in two ASIAA patients 


Normal sensation $ 


Altered sensation $ 


Number of dermatomes 


Right Side 


Left Side 


Start of After 10 Start of After 10 
training months of training months of 
training training 


FIGURE 20. Sensory improvement after neurorehabilitation training. A: average sensory improvement 
(mean + SE over all patients) after 10 mo training. B: example of improvement on Zone of Partial Preservation 
for sensory evaluation for two patients. [From Donati et al. (205). ] 
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nals were generated either when the avatar body walked on 
a virtual surface, or when the patients walked with the help 
of the robotic devices. In the latter case, pressure sensors 
applied to the plantar surface of the robotic feet were re- 
sponsible for generating signals depicting the feet’s contact 
with the ground during bipedal walking. Closing the con- 
trol loop with this haptic display led patients to experience 
vivid lower limb phantom sensations, which included the 
illusion of experiencing leg movements even when they 
were operating the avatar body while remaining immobile 


themselves. Moreover, using the information delivered by 
the haptic display applied to the skin surface of their fore- 
arms, six out of eight patients could discriminate between 
three different types of surface in which the avatar body 
walked (e.g., sand, grass, and asphalt). 


Despite being completely paraplegic, immobile from the 
level of the spinal cord lesion down, lesions ranging from 
(T4-T11), since the day of their spinal cord lesions (3-13 
years earlier), and lacking any somatic sensation below the 
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FIGURE 21. Lower limb motor recovery. A: details of the EMG recording procedure in SCI patients. A7: raw 
EMG for the right gluteus maximus muscle for patient P1 is shown at the top of the topmost graph. The lower 
part of this graph depicts the envelope of the raw EMG, after the signal was rectified and low pass filtered at 
3 Hz. Gray shaded areas represent periods where the patient was instructed to move the right leg, while the 
blue shaded areas indicate periods of left leg movement. Red areas indicate periods where patients were 
instructed to relax both legs. A2: all trials over one session were averaged (mean + SD envelopes are shown) 
and plotted as a function of instruction type (gray envelope = contract right leg; blue = contract left leg; red = 
relax both legs). A3: below the averaged EMG record, light green bars indicate instances in which the voluntary 
muscle contraction (right leg) was significantly different (ttest, P < 0.01) than the baseline (periods where 
she/he was instructed to relax both legs). Dark green bars depict periods in which there was a significant 
difference (P < 0.01) between muscle contraction in the right versus the left leg. B: EMG envelops and t-tests 
for all recording sessions, involving four muscles, for all eight patients: left and right gluteus maximus (GMx) 
and rectus femoris proximal (RFP) muscles. Color convention and figure organization follows the one of A. Data 
were collected after 7 mo of training for all patients and for all but patients P2 and P8 after 12 mo. [From 
Donati et al. (205).] 
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level of the lesion, after a 12 mo period of training with this 
BMI paradigm, all patients exhibited a very significant par- 
tial neurological recovery, which was characterized by the 
following: 1) an average expansion of 5 dermatomes, in 
pinprick, nociceptive sensation, in the zone of partial pres- 
ervation,! (below the level of the lesion) (FIGURE 20, A AND B); 
2) an average 1-2 dermatome expansion in fine touch (FIG- 
URE 20A); 3) significant improvement in proprioception 
and vibration perception below the level of the lesion; 
4) recuperation of voluntary control of multiple muscles 
below the level of the SCI lesion, as measured by EMG 


‘The area of the body, measured in dermatomes and myotomes, 
below the level of the spinal cord lesion, in patients classified as 
having a clinically complete lesion, that remained partially inner- 
vated. 


recordings and direct force measurements. In some cases, 
patients regained the ability to produce multi-joint leg 
movements (FIGURE 21, A AND B); 5) marked improvement 
in the walking index; 6) an improvement in thoracic lumbar 
control (FIGURE 228); and 7) restoration of peristaltic and 
bowel movements, bladder control, and improvement car- 
diovascular function (FIGURE 22C). 


Because of this substantial neurological recovery, 50% of 
the eight patients were upgraded from a complete paraple- 
gia (ASIA A a = 7, ASIA B x = 1) to a partial paraplegia 
classification (ASIA C) at the end of 12 mo of training with 
this BMI-based protocol (FIGURE 22A). Longitudinal anal- 
ysis of EEG recordings obtained from these patients during 
the 12-mo training period reveled that this partial sensory, 
motor, and visceral recovery was paralleled by an expan- 
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FIGURE 22. Clinical and functional improvements. A: patients with ASIA classification improvements: four 
patients changed ASIA classification over the course of the neuro-rehabilitation training, three moved from 
ASIA A to C and one moved from ASIA B to C. ASIA A is characterized by absence of both motor and sensory 
functions in the lowest sacral area; ASIA B by the presence of sensory functions below the neurological level 
of injury, including sacral segments S4-S5 and no motor function is preserved more than three levels below 
the motor level on either side of the body; ASIA C by the presence of voluntary anal sphincter contraction, or 
sacral sensory sparing with sparing of motor function more than three levels below the motor level, majority 
of key muscles have muscle grade less than 319. B: thoracic-lumbar control scale evaluates quantitatively 
motor skill of the thoracolumbar region. Score ranges between O and 65. It has 10 items that consider supine, 
prone, sitting, and standing postures. In the present study, the last item (orthostatic position) was scored O 
due to the limitations of the pathology. C: correlation between average time spent in a standing position in 
orthostatic or gait training (mean + SE, values are average hours per month) and mean frequency for bowel 
function (values calculated per month and z-scored per patient). [From Donati et al. (205).] 
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sion of the representation of lower limbs in their primary 
sensorimotor cortex. Based on these results, Donati et al. 
(205) proposed that a combination of cortical and spinal 
cord plasticity, triggered by chronic use of a BMI that pro- 
vided rich visuo-tactile feedback, may have rekindled re- 
maining axons that survived the original spinal cord injury. 


Overall, the study by Donati et al. (205) raises the concrete 
possibility that the future goals of BMI research may in- 
clude the possibility of creating therapeutic procedures 
aimed at inducing some degree of neurological recovery in 
patients suffering from incomplete SCIs. As such, BMIs may 
become a true neurorehabilitation paradigm for these pa- 
tients, instead of a mere assistive technology. 


Xill. CONCLUSION 


After a decade and a half of intense development, BMI 
research is currently witnessing a very rapid growth to- 
wards a broad range of potential clinical applications. This 
trend was originally driven by the expectation that BMIs 
may provide fundamental assistive tools for people who 
suffer from motor and/or sensory deficits. Recently, this 
expectation has been upgraded to reflect the possibility that 
BMIs may also become a new neurorehabilitation therapy 
that takes advantage of the phenomenon of brain plasticity 
to induce partial neurological recovery in severely disabled 
patients. 
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over the internet. Like magicians, we will simply think and send messages, 
move objects, feel the thoughts and emotions of others, 
and control exoskeletons with superpowers.” 


Michio Kaku, physicist and futurologist talking about Miguel Nicolelis’ 
book Beyond Boundaries: The New Neuroscience of Connecting Brains 
with Machines - and How It Will Change Our Lives 


